Mode d’emploi
- Copiez la commande et examinez le paquet avant de l’installer dans votre environnement agent.
- Consultez le document original pour les outils, identifiants et environnements requis.
- Décrivez les entrées et le livrable souhaité ; comparez la sortie aux sources.
npx skills add https://github.com/k8s-servant/k8s-skills --skill k8s-istio-troubleshootingAvant de commencer
- L’installation n’inclut ni abonnement au modèle, ni quota API, ni accès aux fichiers.
- Étoiles et installations reflètent l’activité, pas un test indépendant de qualité ou de sécurité.
Rendu depuis le dépôt source en conservant titres, exemples, code, tableaux, liens et images.
Istio troubleshooting
Find out why requests fail or behave unexpectedly in an Istio mesh, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.
In sidecar mode every meshed pod has an Envoy container, istio-proxy, injected when its namespace has istio-injection=enabled (or an istio.io/rev label). The client's sidecar routes by VirtualServices and DestinationRules (subsets, load balancing, TLS); the server's sidecar enforces PeerAuthentication (mTLS) and AuthorizationPolicies. Each sidecar writes an access log line per request with a response flag that says what Envoy did. In ambient mode the node's ztunnel handles L4 and mTLS and an optional waypoint proxy handles L7.
The commands use placeholders: my-namespace, my-app (the server and its Service), my-web (the client) and my-pod. Replace them with the real names. The built-in view role cannot read Istio's resources (Istio adds nothing to it, and its own reader role also grants Secrets); if they are forbidden, say so and ask the user for them. Never read Secrets: not certificates, not istio-ca-secret, not gateway TLS Secrets. For that reason do not run istioctl analyze against the cluster (it reads Secrets), do not use istioctl proxy-config secret, and do not fetch Envoy's full config_dump or /certs.
When to use
- A request inside the mesh, or through an Istio gateway, returns 403, 404, 503 or a
reset, while the app itself is fine.
- Routing rules (weights, subsets, headers) or policies do not take effect.
- A pod has no
istio-proxycontainer although its namespace is in the mesh. - istiod is down, or proxies do not get configuration.
Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:
- The Service has no endpoints, or the pods are not Ready:
k8s-network-troubleshooting
and k8s-workload-troubleshooting.
- A Kubernetes Gateway API route that Istio does not accept:
k8s-gateway-api-troubleshooting (the objects' status) and the reference here.
- What an Envoy response flag, reset reason or cluster statistic means in general:
k8s-envoy-troubleshooting (the proxy itself; Istio writes its config).
Investigate
Work through the steps in order. Stop when you have the cause and its evidence.
1. Check the mesh membership of the client and the server: the namespace labels, and whether each pod has an istio-proxy container.
kubectl get namespace my-namespace --show-labels
kubectl get pods -n my-namespace -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.spec.containers[*].name}{" "}{.spec.initContainers[*].name}{"\n"}{end}'
kubectl get pods -n istio-system2. Read the client's error and both sidecars' access logs. The response flag after the status code says what went wrong.
kubectl logs -n my-namespace deployment/my-web -c istio-proxy --tail=20
kubectl logs -n my-namespace deployment/my-app -c istio-proxy --tail=20A log line reads "GET / HTTP/1.1" 503 UH no_healthy_upstream ... outbound|80|v2|my-app...: status, response flag, details, and the Envoy cluster (direction, port, subset, host). Access logs must be on (meshConfig.accessLogFile, or a Telemetry resource); if they are empty, say so.
3. Read the Istio resources for the host and the server.
kubectl get virtualservices,destinationrules,gateways,serviceentries,sidecars -A
kubectl get authorizationpolicies,peerauthentications,requestauthentications -A
kubectl get virtualservice my-app -n my-namespace -o yaml
kubectl get destinationrule my-app -n my-namespace -o yaml
kubectl get pods -n my-namespace --show-labels4. If the cause is still open, ask the proxy itself through its admin API, with GET only, on the pod that logs the error. This runs a program in the user's pod, so ask first:
Ask first: the agent runs this only after the user approves it.
kubectl exec my-pod -n my-namespace -c istio-proxy -- pilot-agent request GET clusters
kubectl exec my-pod -n my-namespace -c istio-proxy -- pilot-agent request GET statsclusters lists each Envoy cluster with its endpoints and their health. Never send any method other than GET, and never request config_dump without a resource filter or certs.
5. Open the matching reference and follow its steps:
| What you see | Reference |
|---|---|
503 UH, UF, URX, 404 NR, upstream connect error, routing that does not apply, subsets, ServiceEntries, gateways | references/routing.md |
403 RBAC: access denied, Connection reset by peer, filter_chain_not_found, mTLS, injection, istiod, ambient | references/security-and-injection.md |
Read the results
What the common findings mean (Istio 1.31):
- **403 with the body
RBAC: access denied, and the server's sidecar logs `403 -
rbacaccessdeniedmatchedpolicy[...]**: an AuthorizationPolicy denies it. [none]` means an ALLOW policy selects the workload and no rule matched this request (a principal, namespace, path or method that is not listed).
- 503 `UH` `no_healthy_upstream` on the client's sidecar, with a subset in the
cluster name (outbound|80|v2|...): the route points at a DestinationRule subset whose labels match no ready pod.
- 404 `NR` (no route) on the client or a gateway: no VirtualService or route matches
the host, path or port.
- `upstream connect error or disconnect/reset before headers. reset reason: ...`
(with UF or URX): Envoy could not connect to, or keep, the upstream: the app is down, the port is wrong, or the mTLS settings of client and server disagree.
- **The client gets
Connection reset by peer, and the server's sidecar logs `NR
filterchainnot_found**: the server requires mTLS (PeerAuthentication STRICT`) and the client sent plaintext, usually because it has no sidecar.
State the cause in one sentence, then the evidence you saw (the access log line and its flag, the resource and the field), then the proposed fix.
Proposed fix
Show each fix to the user before running it: what it changes and how to undo it. Istio resources change traffic for every client of a host, and policies change who may talk to whom: never create, change or delete them yourself; hand the change to the user as a proposed fix, in the source they are deployed from. Restarting a workload so it gets a sidecar is reversible, so it goes under "Ask first". Write each undo with the values you found.
Restart a workload after its namespace was added to the mesh, so its pods get a sidecar:
Ask first: the agent runs this only after the user approves it.
kubectl rollout restart deployment/my-web -n my-namespace
# Undo: kubectl rollout undo deployment/my-web -n my-namespaceSources
Checked against the Istio 1.31 documentation and source (1.31.3):
security problems and injection problems.
- Diagnostic tools and
security and authorization policy.
and ambient mode.
- Helm installation.
- Envoy's response flags
and admin interface.
istioctl analyzereads Secrets (its gateway analyzer:
secret.go), which is why this skill does not run it. The log lines quoted here were seen on a Kubernetes 1.37 kind cluster with Istio 1.31.3 in sidecar mode.