k8s-servant/k8s-skills

k8s-triage

Use when something is wrong in a cluster and it is not yet clear where: "the cluster is broken", "a lot of pods are failing", "users say the app is down", "something is off after the upgrade", or any problem report that names no workload.

查看源码

使用方法

  1. 复制安装命令,在自己的智能体环境安装前先检查源码包。
  2. 从原始文档核对所需工具、凭据和支持的宿主。
  3. 说明输入材料与期望交付物,并对照来源检查输出。
npx skills add https://github.com/k8s-servant/k8s-skills --skill k8s-triage

开始前需了解

  • 安装不包含模型订阅、API 额度或访问文件的权限。
  • 仓库星数和安装量反映活跃情况,不等于独立质量或安全测试。
仓库原始内容

按源仓库内容呈现,保留标题、案例、代码、表格、链接以及原文引用的演示图片。

Kubernetes triage

Find out where a cluster is unhealthy when nobody knows yet, rank the problems, and hand each one to the right place. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

Triage is a first look, not a root-cause analysis. For each problem, report it, show the evidence, propose a first fix, and name the skill that investigates it fully. Do not dig deeper than the short first look this skill describes.

The commands use placeholders: my-namespace, my-app, my-pod and my-node. Replace them with the real names. Never read Secrets. If output contains passwords, tokens or keys, do not repeat them.

When to use

  • The user reports a problem without naming a workload: "the cluster is broken",

"everything is slow", "lots of pods are failing", "users say the app is down".

  • Something changed cluster-wide (an upgrade, a new node pool, a network change) and

"something is off".

  • You were asked for a general health check of a cluster.

When the user names one workload that is unhealthy, start with k8s-workload-troubleshooting instead. When nothing is broken and the user wants to know whether the cluster is in good shape (good practice, what drifted, leftovers), use k8s-cluster-review instead.

Hand-offs, by skill name. For each issue you find, name the skill that goes deeper. If it is not installed, give the issue, its evidence, a first fix and the short first look below, then tell the user which skill would help, and stop there:

  • Pods crash, restart, stay Pending, fail probes, or a rollout or Job is stuck:

k8s-workload-troubleshooting.

  • A node is NotReady, under memory, disk or PID pressure, cordoned, or pods are being

evicted: k8s-node-troubleshooting.

  • Pods are healthy but traffic does not arrive (Service with no endpoints, DNS,

NetworkPolicy, Ingress): k8s-network-troubleshooting.

  • A PersistentVolumeClaim is Pending, or volumes fail to attach or mount:

k8s-storage-troubleshooting.

  • The API server refuses requests (failed calling webhook, an APIService that is not

True, a namespace stuck Terminating): k8s-api-troubleshooting.

  • On AKS, the error comes from Azure (SyncLoadBalancerFailed, a ProvisioningFailed

with an Azure error code, AADSTS errors, a cluster or node pool Failed): k8s-aks-troubleshooting.

Investigate

Work through the steps in order. They are cheap and cluster-wide; do all of them before you rank anything.

1. Check what you can see. A user with access to one namespace cannot list nodes or other namespaces. Know this before you read any result.

bash
kubectl auth can-i list nodes
kubectl auth can-i list pods --all-namespaces
kubectl auth can-i list events --all-namespaces

If a check says no, or a command below fails with Forbidden, say exactly what you could not see, and narrow the sweep to the namespaces you can read. Start with the namespace your kubeconfig points at: commands without -n use it, and users with access to one namespace usually have it set.

bash
kubectl auth can-i list pods
kubectl get pods
kubectl auth can-i list pods -n my-namespace

If none of these works and you know no namespace, ask the user which one to look at. Never report something you could not see as healthy.

2. Nodes. Look at STATUS (Ready, NotReady, and SchedulingDisabled for a cordoned node), then describe every node that is not plainly Ready, and read its Conditions, Taints and Allocated resources.

bash
kubectl get nodes -o wide
kubectl describe node my-node

3. Pods that are not running. Across every namespace you can read:

bash
kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
kubectl get pods -A | grep -v -E "Running|Completed"
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount'

The first command finds Pending, Failed and Unknown pods. The second also catches pods whose phase is Running but whose STATUS column says CrashLoopBackOff, ImagePullBackOff, Error or Terminating. The third puts the most restarted pods last. Also look for Running pods that are not ready, such as 0/1 in the READY column.

When a node is not Ready, list the pods on it:

bash
kubectl get pods -A --field-selector=spec.nodeName=my-node -o wide

4. Warning events. For each one, check whether the object it is about is still unhealthy now. Only those count as issues; a Warning about an object that has since recovered is history, even if it is recent (a new cluster has several from start-up).

bash
kubectl events -A --types=Warning

5. CPU and memory. kubectl top works only when metrics-server is installed and running. If it fails, write "metrics not available" in the overall status and go on; Allocated resources from step 2 still shows requests and limits against capacity.

bash
kubectl top nodes
kubectl top pods -A --sort-by=memory
kubectl top pods -A --sort-by=cpu

6. Controllers short of replicas. A Deployment, StatefulSet or DaemonSet whose ready count is below the desired count is affected even when its pods look fine one by one.

bash
kubectl get deployments -A
kubectl get statefulsets -A
kubectl get daemonsets -A

7. PersistentVolumeClaims and Services, only if steps 3 and 4 point there (pods Pending with volume events, or an app that is up but unreachable):

bash
kubectl get pvc -A
kubectl get endpointslices -n my-namespace

8. Pick the area of each issue and its next skill, from this table:

What you seeTierNext skill
Node NotReady or Unknown, a pressure condition True, NetworkUnavailable, SchedulingDisabled1k8s-node-troubleshooting
CrashLoopBackOff, OOMKilled, Error, ErrImagePull, ImagePullBackOff, CreateContainerConfigError2k8s-workload-troubleshooting
Pending, Running but not ready, a controller short of replicas3k8s-workload-troubleshooting (or k8s-storage-troubleshooting for an unbound PVC)
Warning events about objects that are still unhealthy, a node near its memory or CPU capacity, pods at their limits4the area the event belongs to
Service with no ready endpoints while its pods are healthy, DNS failures3k8s-network-troubleshooting
CoreDNS pods crashing or not Ready, CoreDNS errors in its logs2k8s-coredns-troubleshooting
FailedCreate events that name a webhook, an APIService that is not True, a namespace stuck Terminating3k8s-api-troubleshooting
On AKS: SyncLoadBalancerFailed, Azure error codes in ProvisioningFailed or FailedAttachVolume, AADSTS errors in logs3k8s-aks-troubleshooting

A short first look inside a container can confirm a network issue. Ask first: show the command, and run it only after the user agrees (pick a running pod in the namespace that has nslookup; many minimal images do not):

Ask first: the agent runs this only after the user approves it.
bash
kubectl exec my-pod -n my-namespace -- nslookup my-app.my-namespace.svc.cluster.local

Read the results

Rank by severity. Report issues in this order, most urgent first:

  1. Nodes. NotReady or Unknown, MemoryPressure, DiskPressure or PIDPressure

True, NetworkUnavailable, cordoned (SchedulingDisabled).

  1. Crashing pods. CrashLoopBackOff, OOMKilled, Error, ErrImagePull or

ImagePullBackOff, CreateContainerConfigError.

  1. Pods that do not become ready. Pending, Running but not ready, controllers

with fewer ready replicas than desired, Services without ready endpoints.

  1. Early warnings. Warning events about objects that are still unhealthy, and

resource pressure that has not broken anything yet (a node above about 90% of its memory, pods at their CPU or memory limit).

Within a tier, more affected nodes or pods rank higher. One cause, one issue: when a node problem explains pod symptoms (the pods on a NotReady node show Unknown, Terminating, or are being recreated elsewhere), report the node issue once and list the pods as its evidence. Do not report those pods as separate application failures.

Read nodes. Ready is Unknown when the node controller has not heard from the node within its grace period (50 seconds by default): the kubelet is down or the node is unreachable. Ready is False when the kubelet reports the node is not healthy. A pressure condition True means the kubelet may evict pods from that node. The taints node.kubernetes.io/not-ready, node.kubernetes.io/unreachable and node.kubernetes.io/unschedulable (a cordoned node) show the same state to the scheduler. By default, pods on a NotReady or unreachable node are evicted after five minutes; until then they look stuck, and pods of a StatefulSet can stay Terminating.

Read events. Typical reasons: FailedScheduling, BackOff, Failed (image pull), Unhealthy (probes), FailedMount, Evicted, NodeNotReady, OOMKilling. An event is evidence only when its object is unhealthy now: a Running, ready pod that once failed a readiness probe is not an issue.

Write the report in this shape:

  1. Overall status. Nodes (how many, how many Ready), pods (how many are not

healthy, in how many namespaces), metrics (available or not), and what you could not see because of access.

  1. Issues, ranked. For each: one sentence for the issue, the evidence you saw (the

status, the condition, the event reason and message, the number), the next skill, and a first fix in the format under Proposed fix.

  1. When the sweep is clean, do not invent a problem. Say what you checked and that

it looked healthy from where you could see, then ask the user to narrow it down: which app, namespace or URL, what exactly they saw (an error message, a timeout, slowness), and since when.

Proposed fix

Triage proposes first fixes only. Show each one to the user first: what it changes and how to undo it. A reversible fix goes under "Ask first" and runs only after the user says yes. Anything destructive or hard to undo is never run by you; say why and hand it over. Prefer fixing the source the workload is deployed from (manifest, Helm values, GitOps repository) when there is one. Write each undo with the values you found.

Uncordon a node that was left cordoned after maintenance and is otherwise healthy:

Ask first: the agent runs this only after the user approves it.
bash
kubectl uncordon my-node
# Undo: kubectl cordon my-node

Roll back a Deployment whose pods started failing right after a rollout. First note the current revision with kubectl rollout history deployment/my-app -n my-namespace; the undo below returns to it by number (here revision 3), so it holds even if another rollout happens in between:

Ask first: the agent runs this only after the user approves it.
bash
kubectl rollout undo deployment/my-app -n my-namespace
# Undo: kubectl rollout undo deployment/my-app -n my-namespace --to-revision=3

Force-delete a pod stuck Terminating on a node that is gone for good. Never run this yourself: for a StatefulSet pod it can start a second copy while the first still runs on a node that is only unreachable. A human must first confirm the node is really gone.

Proposed fix: review before running. The agent does not run this.
bash
kubectl delete pod my-pod -n my-namespace --force --grace-period=0
# Undo: none; the pod's controller recreates it on another node

For a node that stays NotReady, a workload that keeps crashing, a Service without endpoints or a stuck volume, the fix needs the deeper skill named in the issue.

Sources

Checked against the Kubernetes 1.37 documentation:

Node Status: conditions, the 50-second grace period before Unknown, cordoned nodes shown as SchedulingDisabled, eviction five minutes after Unknown.

pressure conditions and eviction signals.

the node.kubernetes.io/* taints, the default 300-second tolerations, pods still running on a partitioned node.

pod phases (Unknown when the node cannot be reached), Terminating, CrashLoopBackOff.

status.phase and spec.nodeName for pods.

--types, -A.

and kubectl top: kubectl top needs metrics-server.

listing and describing nodes.

the view role and kubectl auth can-i.

来自同一仓库

更多 Skills

全部 Skills
k8s-servant
社区

k8s-api-troubleshooting

Use when the Kubernetes API server itself refuses or fails a request, before any controller acts on it: "failed calling webhook", "admission webhook ... denied the request", "ValidatingAdmissionPolicy ... denied request", a ReplicaSet with FailedCreate and no pods, an APIService "False (ServiceNotFound)", "FailedDiscoveryCheck" or "MissingEndpoints", "unable to retrieve the complete list of server APIs", a namespace stuck in Terminating, "no matches for kind" or "ensure CRDs are installed first"

1小时安装量
1
GitHub Stars
38
最近更新
11月20日
k8s-servant
社区

k8s-argocd-troubleshooting

Use when Argo CD does not deploy or update an application in Kubernetes: an Application that is OutOfSync, Unknown, Degraded, Missing or Progressing, a ComparisonError ("app path does not exist", "unable to resolve ... to a commit SHA"), an InvalidSpecError ("do not match any of the allowed destinations in project"), a sync that fails ("one or more objects failed to apply"), auto-sync that never runs, resources that stay OutOfSync after a sync, or ApplicationSets that generate nothing. Investiga

1小时安装量
1
GitHub Stars
38
最近更新
11月20日
k8s-servant
社区

k8s-cert-manager-troubleshooting

Use when cert-manager does not issue or renew a certificate in Kubernetes: a Certificate that stays READY False, a CertificateRequest that is Pending, Denied or Failed, an Issuer or ClusterIssuer that is not Ready ("Error getting keypair for CA issuer"), "Referenced \"Issuer\" not found", ACME Orders or Challenges stuck pending (HTTP-01, DNS-01, rate limits), a certificate that expired or did not renew, or "failed calling webhook \"webhook.cert-manager.io\"" when applying resources. Investigates

1小时安装量
1
GitHub Stars
38
最近更新
11月20日
k8s-servant
社区

k8s-cilium-troubleshooting

Use when networking fails on a Kubernetes cluster that runs Cilium as its CNI: traffic between pods dropped or timing out after a CiliumNetworkPolicy or CiliumClusterwideNetworkPolicy, "Policy denied" drops in Hubble, DNS that stops resolving after an egress policy, HTTP 403 "Access denied" from an L7 rule, toFQDNs rules that do not match, pods stuck ContainerCreating with "unable to connect to Cilium agent", Cilium agent or operator pods not ready, CiliumEndpoints not ready, or IPAM exhaustion.

1小时安装量
1
GitHub Stars
38
最近更新
11月20日