使用方法
- 复制安装命令,在自己的智能体环境安装前先检查源码包。
- 从原始文档核对所需工具、凭据和支持的宿主。
- 说明输入材料与期望交付物,并对照来源检查输出。
npx skills add https://github.com/k8s-servant/k8s-skills --skill k8s-prometheus-troubleshooting开始前需了解
- 安装不包含模型订阅、API 额度或访问文件的权限。
- 仓库星数和安装量反映活跃情况,不等于独立质量或安全测试。
按源仓库内容呈现,保留标题、案例、代码、表格、链接以及原文引用的演示图片。
Prometheus troubleshooting
Find out why Prometheus does not scrape a target, does not load a rule, or does not alert, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
With the Prometheus Operator, a Prometheus custom resource selects ServiceMonitors, PodMonitors, Probes, ScrapeConfigs and PrometheusRules by label and namespace, and the operator turns them into Prometheus' configuration. A ServiceMonitor selects Services by label and names one of their ports. Something can go wrong at each step: the selection, the Service, the port, the scrape itself.
The commands use placeholders: monitoring (where Prometheus runs), my-prometheus (the Prometheus resource), my-namespace, my-app (the ServiceMonitor and its Service) and my-rules. Replace them with the real names. Prometheus' HTTP API is reached through the API server's service proxy on the governing Service prometheus-operated (port name web); this needs get on services/proxy, which the built-in view role does not have. If the proxy is forbidden, say so and ask the user for the Targets page of the Prometheus UI.
Never read Secrets. The operator stores the generated Prometheus and Alertmanager configuration in Secrets (prometheus-<name>, alertmanager-<name>-generated), and the upstream troubleshooting guide decodes them. Do not: use the selection rules, the operator's events and logs, and Prometheus' own API instead. For the same reason never call Prometheus' /api/v1/status/config or Alertmanager's /api/v2/status: they return that generated configuration (credentials masked, but webhook URLs and routes not). Access to the service proxy covers the whole Service, so this rule is yours to keep.
When to use
- A new ServiceMonitor or PodMonitor shows no targets, or the job is missing.
- Targets are
down. - A PrometheusRule's alerts do not appear, do not fire, or do not reach Alertmanager.
- The
PrometheusorAlertmanagerresource is notAvailable, or its pods crash.
Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:
- The target's pods are not running or not Ready:
k8s-workload-troubleshooting. - The target is up but a NetworkPolicy or DNS blocks the scrape:
k8s-network-troubleshooting.
- Prometheus' volume is full or will not bind:
k8s-storage-troubleshooting.
Investigate
Work through the steps in order. Stop when you have the cause and its evidence.
1. Read the Prometheus resource: its status, and which monitors and rules it selects.
kubectl get prometheuses -A
kubectl get prometheus my-prometheus -n monitoring -o jsonpath='{.spec.serviceMonitorSelector}{"\n"}{.spec.serviceMonitorNamespaceSelector}{"\n"}{.spec.podMonitorSelector}{"\n"}{.spec.ruleSelector}{"\n"}{.spec.ruleNamespaceSelector}{"\n"}{.status.conditions}{"\n"}'2. Read the ServiceMonitor and compare it with the selection and with the Service.
kubectl get servicemonitors,podmonitors -A
kubectl get servicemonitor my-app -n my-namespace -o yaml
kubectl get service my-app -n my-namespace -o jsonpath='{.metadata.labels}{"\n"}{.spec.ports}{"\n"}'
kubectl get endpointslices -n my-namespace -l kubernetes.io/service-name=my-app
kubectl get namespace my-namespace --show-labels3. Ask Prometheus what it scrapes: the jobs it has, the active targets with their health and last error, and the targets it dropped.
kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/scrape_pools'
kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/targets?state=active' | jq -r '.data.activeTargets[] | [.labels.job, .health, .scrapeUrl, .lastError] | @tsv'
kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/targets?state=dropped' | jq -r '.data.droppedTargets[].discoveredLabels | [.__meta_kubernetes_service_name, .__meta_kubernetes_endpoint_port_name, .__address__] | @tsv'A ServiceMonitor appears as the job serviceMonitor/<namespace>/<name>/<endpoint index>.
4. Read the operator's events and logs. The operator rejects invalid objects with an event and logs objects it skips.
kubectl events -n my-namespace --for servicemonitor/my-app
kubectl get deployments -A -l app.kubernetes.io/name=prometheus-operator
kubectl logs -n monitoring deployment/prometheus-operator --tail=200 | grep -i -E "skipping|rejected|my-app"5. Open the matching reference and follow its steps:
| What you see | Reference |
|---|---|
| No job or no targets for a ServiceMonitor or PodMonitor, dropped targets | references/discovery.md |
Targets down: HTTP status, connection refused, timeouts, body size, content type | references/targets-down.md |
| A PrometheusRule not loaded, alerts not firing or not reaching Alertmanager | references/rules-and-alerts.md |
| The Prometheus resource not Available, pods OOMKilled, disk full, ingestion errors | references/prometheus-health.md |
Read the results
What the common findings mean (Prometheus Operator 0.94, Prometheus 3.14):
- No job `serviceMonitor/<ns>/<name>/...` in `scrape_pools`: the Prometheus does not
select the ServiceMonitor. Its labels do not match serviceMonitorSelector, or its namespace does not match serviceMonitorNamespaceSelector. An empty selector ({}) matches everything; an unset (null) selector matches nothing for monitors and rules, and only Prometheus' own namespace for the namespace selectors.
- The job exists, no active targets, a dropped target: the ServiceMonitor matches a
Service, but its endpoints[].port is not the name of one of the Service's ports (it must be the port's name, not its number), or the Service has no ready endpoints. The dropped target's __meta_kubernetes_endpoint_port_name shows the real port name.
- The job exists, no targets at all, nothing dropped:
spec.selectormatches no
Service, or namespaceSelector leaves the Service's namespace out.
- A target `down` with `server returned HTTP status 404 Not Found`: the path is
wrong (endpoints[].path, default /metrics). Other lastError values are in references/targets-down.md.
- **An event
InvalidConfiguration, `"<name>" was rejected due to invalid
configuration: ...`**: the operator refused the object; the message says why.
State the cause in one sentence, then the evidence you saw (the selector and labels, the port names, the target's health and error), then the proposed fix.
Proposed fix
Show each fix to the user before running it: what it changes and how to undo it. Patching a ServiceMonitor, PodMonitor or PrometheusRule (labels, port, path) is reversible, so it goes under "Ask first" and runs only after the user says yes. Changing the Prometheus or Alertmanager resource (selectors, resources, retention), Alertmanager's configuration and anything with Secrets are never run by you: hand them to the user as proposed fixes. Monitors usually come from the app's Helm chart or manifests; fix them there too. Write each undo with the values you found.
Give a ServiceMonitor the label the Prometheus selects:
Ask first: the agent runs this only after the user approves it.
kubectl label servicemonitor my-app -n my-namespace release=my-prometheus
# Undo: kubectl label servicemonitor my-app -n my-namespace release-Point a ServiceMonitor at the Service's real port name:
Ask first: the agent runs this only after the user approves it.
kubectl patch servicemonitor my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "http"}]'
# Undo: kubectl patch servicemonitor my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "metrics"}]'Prometheus picks up the change by itself within a minute or two; check scrape_pools and the targets again.
Sources
Checked against the Prometheus Operator 0.94 and Prometheus 3.14 documentation and source:
selection, port names, events; its step that decodes the configuration Secret is left out here.
- API reference: selector
semantics (empty versus null), ServiceMonitor endpoints.
- Prometheus HTTP API
(targets, scrape pools, rules, alerts, TSDB status), storage and the 3.0 migration guide.
- Manually constructing API server proxy URLs.
- Messages from the source:
resource_selector.go, operator.go, scrape.go. The behaviour and messages quoted here were also seen on a Kubernetes 1.37 kind cluster with Prometheus Operator 0.94.2 and Prometheus 3.14.0.