k8s-servant/k8s-skills

k8s-prometheus-troubleshooting

Use when Prometheus on Kubernetes does not collect or alert as expected: a ServiceMonitor or PodMonitor whose targets never appear, targets that are down ("server returned HTTP status 404 Not Found", "connection refused", "context deadline exceeded"), a Pro…

Zobacz źródło

Jak używać

  1. Skopiuj polecenie instalacji i sprawdź źródła przed instalacją we własnym środowisku agenta.
  2. W oryginale sprawdź wymagane narzędzia, dane dostępowe i obsługiwane środowiska.
  3. Opisz dane wejściowe i oczekiwany rezultat; porównaj wynik ze źródłami.
npx skills add https://github.com/k8s-servant/k8s-skills --skill k8s-prometheus-troubleshooting

Przed rozpoczęciem

  • Instalacja nie obejmuje subskrypcji modelu, limitu API ani uprawnień do plików.
  • Gwiazdki i instalacje pokazują aktywność, a nie niezależny test jakości lub bezpieczeństwa.
Oryginalny dokument Skill

Treść z repozytorium z zachowaniem nagłówków, przykładów, kodu, tabel, linków i obrazów.

Prometheus troubleshooting

Find out why Prometheus does not scrape a target, does not load a rule, or does not alert, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

With the Prometheus Operator, a Prometheus custom resource selects ServiceMonitors, PodMonitors, Probes, ScrapeConfigs and PrometheusRules by label and namespace, and the operator turns them into Prometheus' configuration. A ServiceMonitor selects Services by label and names one of their ports. Something can go wrong at each step: the selection, the Service, the port, the scrape itself.

The commands use placeholders: monitoring (where Prometheus runs), my-prometheus (the Prometheus resource), my-namespace, my-app (the ServiceMonitor and its Service) and my-rules. Replace them with the real names. Prometheus' HTTP API is reached through the API server's service proxy on the governing Service prometheus-operated (port name web); this needs get on services/proxy, which the built-in view role does not have. If the proxy is forbidden, say so and ask the user for the Targets page of the Prometheus UI.

Never read Secrets. The operator stores the generated Prometheus and Alertmanager configuration in Secrets (prometheus-<name>, alertmanager-<name>-generated), and the upstream troubleshooting guide decodes them. Do not: use the selection rules, the operator's events and logs, and Prometheus' own API instead. For the same reason never call Prometheus' /api/v1/status/config or Alertmanager's /api/v2/status: they return that generated configuration (credentials masked, but webhook URLs and routes not). Access to the service proxy covers the whole Service, so this rule is yours to keep.

When to use

  • A new ServiceMonitor or PodMonitor shows no targets, or the job is missing.
  • Targets are down.
  • A PrometheusRule's alerts do not appear, do not fire, or do not reach Alertmanager.
  • The Prometheus or Alertmanager resource is not Available, or its pods crash.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • The target's pods are not running or not Ready: k8s-workload-troubleshooting.
  • The target is up but a NetworkPolicy or DNS blocks the scrape:

k8s-network-troubleshooting.

  • Prometheus' volume is full or will not bind: k8s-storage-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. Read the Prometheus resource: its status, and which monitors and rules it selects.

bash
kubectl get prometheuses -A
kubectl get prometheus my-prometheus -n monitoring -o jsonpath='{.spec.serviceMonitorSelector}{"\n"}{.spec.serviceMonitorNamespaceSelector}{"\n"}{.spec.podMonitorSelector}{"\n"}{.spec.ruleSelector}{"\n"}{.spec.ruleNamespaceSelector}{"\n"}{.status.conditions}{"\n"}'

2. Read the ServiceMonitor and compare it with the selection and with the Service.

bash
kubectl get servicemonitors,podmonitors -A
kubectl get servicemonitor my-app -n my-namespace -o yaml
kubectl get service my-app -n my-namespace -o jsonpath='{.metadata.labels}{"\n"}{.spec.ports}{"\n"}'
kubectl get endpointslices -n my-namespace -l kubernetes.io/service-name=my-app
kubectl get namespace my-namespace --show-labels

3. Ask Prometheus what it scrapes: the jobs it has, the active targets with their health and last error, and the targets it dropped.

bash
kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/scrape_pools'
kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/targets?state=active' | jq -r '.data.activeTargets[] | [.labels.job, .health, .scrapeUrl, .lastError] | @tsv'
kubectl get --raw '/api/v1/namespaces/monitoring/services/prometheus-operated:web/proxy/api/v1/targets?state=dropped' | jq -r '.data.droppedTargets[].discoveredLabels | [.__meta_kubernetes_service_name, .__meta_kubernetes_endpoint_port_name, .__address__] | @tsv'

A ServiceMonitor appears as the job serviceMonitor/<namespace>/<name>/<endpoint index>.

4. Read the operator's events and logs. The operator rejects invalid objects with an event and logs objects it skips.

bash
kubectl events -n my-namespace --for servicemonitor/my-app
kubectl get deployments -A -l app.kubernetes.io/name=prometheus-operator
kubectl logs -n monitoring deployment/prometheus-operator --tail=200 | grep -i -E "skipping|rejected|my-app"

5. Open the matching reference and follow its steps:

What you seeReference
No job or no targets for a ServiceMonitor or PodMonitor, dropped targetsreferences/discovery.md
Targets down: HTTP status, connection refused, timeouts, body size, content typereferences/targets-down.md
A PrometheusRule not loaded, alerts not firing or not reaching Alertmanagerreferences/rules-and-alerts.md
The Prometheus resource not Available, pods OOMKilled, disk full, ingestion errorsreferences/prometheus-health.md

Read the results

What the common findings mean (Prometheus Operator 0.94, Prometheus 3.14):

  • No job `serviceMonitor/<ns>/<name>/...` in `scrape_pools`: the Prometheus does not

select the ServiceMonitor. Its labels do not match serviceMonitorSelector, or its namespace does not match serviceMonitorNamespaceSelector. An empty selector ({}) matches everything; an unset (null) selector matches nothing for monitors and rules, and only Prometheus' own namespace for the namespace selectors.

  • The job exists, no active targets, a dropped target: the ServiceMonitor matches a

Service, but its endpoints[].port is not the name of one of the Service's ports (it must be the port's name, not its number), or the Service has no ready endpoints. The dropped target's __meta_kubernetes_endpoint_port_name shows the real port name.

  • The job exists, no targets at all, nothing dropped: spec.selector matches no

Service, or namespaceSelector leaves the Service's namespace out.

  • A target `down` with `server returned HTTP status 404 Not Found`: the path is

wrong (endpoints[].path, default /metrics). Other lastError values are in references/targets-down.md.

  • **An event InvalidConfiguration, `"<name>" was rejected due to invalid

configuration: ...`**: the operator refused the object; the message says why.

State the cause in one sentence, then the evidence you saw (the selector and labels, the port names, the target's health and error), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Patching a ServiceMonitor, PodMonitor or PrometheusRule (labels, port, path) is reversible, so it goes under "Ask first" and runs only after the user says yes. Changing the Prometheus or Alertmanager resource (selectors, resources, retention), Alertmanager's configuration and anything with Secrets are never run by you: hand them to the user as proposed fixes. Monitors usually come from the app's Helm chart or manifests; fix them there too. Write each undo with the values you found.

Give a ServiceMonitor the label the Prometheus selects:

Ask first: the agent runs this only after the user approves it.
bash
kubectl label servicemonitor my-app -n my-namespace release=my-prometheus
# Undo: kubectl label servicemonitor my-app -n my-namespace release-

Point a ServiceMonitor at the Service's real port name:

Ask first: the agent runs this only after the user approves it.
bash
kubectl patch servicemonitor my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "http"}]'
# Undo: kubectl patch servicemonitor my-app -n my-namespace --type=json -p '[{"op": "replace", "path": "/spec/endpoints/0/port", "value": "metrics"}]'

Prometheus picks up the change by itself within a minute or two; check scrape_pools and the targets again.

Sources

Checked against the Prometheus Operator 0.94 and Prometheus 3.14 documentation and source:

selection, port names, events; its step that decodes the configuration Secret is left out here.

semantics (empty versus null), ServiceMonitor endpoints.

storage.

(targets, scrape pools, rules, alerts, TSDB status), storage and the 3.0 migration guide.

resource_selector.go, operator.go, scrape.go. The behaviour and messages quoted here were also seen on a Kubernetes 1.37 kind cluster with Prometheus Operator 0.94.2 and Prometheus 3.14.0.

z tego samego repozytorium

Więcej Skills

Wszystkie Skills
k8s-servant
Społeczność

k8s-api-troubleshooting

Use when the Kubernetes API server itself refuses or fails a request, before any controller acts on it: "failed calling webhook", "admission webhook ... denied the request", "ValidatingAdmissionPolicy ... denied request", a ReplicaSet with FailedCreate and no pods, an APIService "False (ServiceNotFound)", "FailedDiscoveryCheck" or "MissingEndpoints", "unable to retrieve the complete list of server APIs", a namespace stuck in Terminating, "no matches for kind" or "ensure CRDs are installed first"

Instalacje / 1h
1
GitHub Stars
38
Aktualizacja
20 lis
k8s-servant
Społeczność

k8s-argocd-troubleshooting

Use when Argo CD does not deploy or update an application in Kubernetes: an Application that is OutOfSync, Unknown, Degraded, Missing or Progressing, a ComparisonError ("app path does not exist", "unable to resolve ... to a commit SHA"), an InvalidSpecError ("do not match any of the allowed destinations in project"), a sync that fails ("one or more objects failed to apply"), auto-sync that never runs, resources that stay OutOfSync after a sync, or ApplicationSets that generate nothing. Investiga

Instalacje / 1h
1
GitHub Stars
38
Aktualizacja
20 lis
k8s-servant
Społeczność

k8s-cert-manager-troubleshooting

Use when cert-manager does not issue or renew a certificate in Kubernetes: a Certificate that stays READY False, a CertificateRequest that is Pending, Denied or Failed, an Issuer or ClusterIssuer that is not Ready ("Error getting keypair for CA issuer"), "Referenced \"Issuer\" not found", ACME Orders or Challenges stuck pending (HTTP-01, DNS-01, rate limits), a certificate that expired or did not renew, or "failed calling webhook \"webhook.cert-manager.io\"" when applying resources. Investigates

Instalacje / 1h
1
GitHub Stars
38
Aktualizacja
20 lis
k8s-servant
Społeczność

k8s-cilium-troubleshooting

Use when networking fails on a Kubernetes cluster that runs Cilium as its CNI: traffic between pods dropped or timing out after a CiliumNetworkPolicy or CiliumClusterwideNetworkPolicy, "Policy denied" drops in Hubble, DNS that stops resolving after an egress policy, HTTP 403 "Access denied" from an L7 rule, toFQDNs rules that do not match, pods stuck ContainerCreating with "unable to connect to Cilium agent", Cilium agent or operator pods not ready, CiliumEndpoints not ready, or IPAM exhaustion.

Instalacje / 1h
1
GitHub Stars
38
Aktualizacja
20 lis