k8s-servant/k8s-skills

k8s-node-troubleshooting

Use when a Kubernetes node is unhealthy or pods will not go onto it: a node NotReady or Unknown, MemoryPressure, DiskPressure or PIDPressure, pods Evicted, pods Pending with "untolerated taint(s)", "Insufficient cpu" or "Insufficient memory" while the node…

Ver código fuente

Cómo usar

  1. Copia el comando y revisa el paquete antes de instalarlo en tu entorno de agente.
  2. Consulta las herramientas, credenciales y entornos necesarios en el documento original.
  3. Describe las entradas y el resultado deseado; contrasta la salida con las fuentes.
npx skills add https://github.com/k8s-servant/k8s-skills --skill k8s-node-troubleshooting

Antes de empezar

  • La instalación no incluye suscripción al modelo, cuota API ni permisos sobre tus archivos.
  • Las estrellas e instalaciones reflejan actividad, no una prueba independiente de calidad o seguridad.
Documento original del Skill

Contenido del repositorio de origen con títulos, ejemplos, código, tablas, enlaces e imágenes preservados.

Kubernetes node troubleshooting

Find out why a node is unhealthy, or why pods will not be scheduled onto it, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

You see the node only through the Kubernetes API. Checks on the node itself (the kubelet, the container runtime, the disk) need someone with access to the machine, so they are always proposed fixes for a human.

The commands use placeholders: my-node, my-namespace, my-app and my-pod. Replace them with the real names. Never read Secrets. If output contains passwords, tokens or keys, do not repeat them.

When to use

  • kubectl get nodes shows a node NotReady, Unknown or SchedulingDisabled.
  • A node reports MemoryPressure, DiskPressure or PIDPressure, or pods were

Evicted.

  • Pods stay Pending and the FailedScheduling event names taints, Insufficient cpu

or Insufficient memory, or unschedulable nodes.

  • The cluster autoscaler does not add a node for Pending pods, or does not remove an

empty one.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • The nodes are healthy and have room, and the pod itself fails (crash, image, probe,

quota): k8s-workload-troubleshooting.

  • A volume cannot attach or mount on the node: k8s-storage-troubleshooting.
  • NetworkUnavailable is True, or the CNI pods on the node fail:

k8s-network-troubleshooting.

  • On AKS, a node pool or the cluster is Failed, or an upgrade or scale fails in Azure

(quota, VM capacity, a drain blocked by a PodDisruptionBudget): k8s-aks-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. List the nodes. STATUS shows Ready, NotReady or Unknown, plus SchedulingDisabled when a node is cordoned.

bash
kubectl get nodes -o wide

2. Describe the node. Read Conditions (status, reason, LastHeartbeatTime), Taints, Unschedulable, Lease (RenewTime), Capacity and Allocatable, Non-terminated Pods with their requests, and Allocated resources.

bash
kubectl describe node my-node

3. See the pods on the node, in every namespace you can read.

bash
kubectl get pods -A -o wide --field-selector spec.nodeName=my-node
kubectl get pods -A --field-selector spec.nodeName=my-node,status.phase=Failed

The second command lists failed pods there, which include Evicted ones.

4. Read the node's events and its heartbeat. Node events are kept in the default namespace; the heartbeat is a Lease named after the node. The built-in view role cannot read Leases; if the second command is forbidden, use LastHeartbeatTime of the conditions in describe node instead.

bash
kubectl get events -A --field-selector involvedObject.kind=Node,involvedObject.name=my-node
kubectl get lease my-node -n kube-node-lease

5. For a Pending pod, read why the scheduler refused it, and what it asks for.

bash
kubectl describe pod my-pod -n my-namespace
kubectl get pod my-pod -n my-namespace -o jsonpath='{.spec.nodeSelector}{"\n"}{.spec.tolerations}{"\n"}{.spec.containers[*].resources.requests}'

6. Look at usage, if the cluster has metrics-server. top shows usage, not requests; the scheduler uses requests. Without metrics-server it fails with "Metrics API not available"; say so and go on.

bash
kubectl top node my-node

7. Open the matching reference and follow its steps:

What you seeReference
NotReady, Unknown, "Kubelet stopped posting node status"references/not-ready.md
MemoryPressure, DiskPressure, PIDPressure, Evicted podsreferences/pressure.md
Insufficient cpu or Insufficient memory, requests near Allocatablereferences/capacity.md
untolerated taint(s), SchedulingDisabled, node(s) were unschedulablereferences/taints.md
Pending pods but no new node, or an empty node that staysreferences/autoscaler.md

Read the results

What the common findings mean (Kubernetes 1.37):

  • `Ready` `Unknown`: the node controller has not heard from the kubelet within

node-monitor-grace-period (50 seconds by default). The kubelet is down, or the node cannot reach the API server. The Lease in kube-node-lease, normally renewed every 10 seconds, stops changing. The node gets the node.kubernetes.io/unreachable taints.

  • `Ready` `False`: the kubelet is running and reports the node is not healthy (for

example the container runtime or the network plugin is not ready). The reason and message say which. The node gets the node.kubernetes.io/not-ready taint.

  • Pods on a `NotReady` node keep their status for a while. Pods tolerate the

not-ready and unreachable NoExecute taints for 300 seconds by default (added by the DefaultTolerationSeconds admission controller), then they are evicted. DaemonSet pods tolerate them without a time limit and are not evicted.

  • `MemoryPressure`, `DiskPressure` or `PIDPressure` `True`: the kubelet crossed an

eviction threshold. It adds a NoSchedule taint for that pressure, frees resources, then evicts pods. Node-pressure eviction does not respect PodDisruptionBudgets.

  • `FailedScheduling` with `0/N nodes are available: ...`: each part counts the nodes

refused for one reason, for example 1 node(s) had untolerated taint(s), 2 Insufficient cpu or 1 node(s) were unschedulable. The text after preemption: says whether evicting lower-priority pods could help.

  • `Insufficient cpu` on an idle node: the scheduler adds up the requests of the pods

on the node and compares them with Allocatable, not with actual use. Allocated resources in describe node shows the requests as a percentage of Allocatable.

  • `SchedulingDisabled`, `Unschedulable: true`: the node is cordoned. New pods do not

go there; pods already there keep running.

State the cause in one sentence, then the evidence you saw (the condition and its message, the taint, the event, the numbers), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Cordoning, uncordoning, adding or removing a NoSchedule taint and labelling a node are reversible, so they go under "Ask first" and run only after the user says yes. Draining or deleting a node, force-deleting pods and anything on the node host are never run by you: hand them to the user as proposed fixes and say why. Prefer fixing the source the workload or the node pool is managed from when there is one. Write each undo with the values you found.

Uncordon a node whose maintenance is over and that is Ready:

Ask first: the agent runs this only after the user approves it.
bash
kubectl uncordon my-node
# Undo: kubectl cordon my-node

Remove a taint left from a maintenance that is over (the key, value and effect exactly as describe node shows them):

Ask first: the agent runs this only after the user approves it.
bash
kubectl taint nodes my-node example.com/maintenance=true:NoSchedule-
# Undo: kubectl taint nodes my-node example.com/maintenance=true:NoSchedule

Check the kubelet on a node that stopped reporting. This runs on the node, which you cannot reach; a person with access to the machine runs it:

Proposed fix: review before running. The agent does not run this.
bash
systemctl status kubelet
journalctl -u kubelet --since "30 min ago" --no-pager | tail -n 50
sudo systemctl restart kubelet
# Undo: nothing to undo for the first two lines, they only read; a restart cannot be undone, the kubelet just starts again

Move the pods off a node that will not come back soon. Draining evicts every pod that is not a DaemonSet pod and deletes emptyDir data, so never run it yourself:

Proposed fix: review before running. The agent does not run this.
bash
kubectl drain my-node --ignore-daemonsets --delete-emptydir-data
# Undo: kubectl uncordon my-node (the evicted pods are recreated by their controllers, not moved back)

Sources

Checked against the Kubernetes 1.37 documentation:

50-second node-monitor-grace-period, Leases in kube-node-lease and their 10-second renewal.

spec.unschedulable.

StatefulSet pods on an unreachable node are not replaced on their own.

and Well-Known Labels, Annotations and Taints: taint effects, the node.kubernetes.io/* taints, taint removal.

DefaultTolerationSeconds and the 300-second default.

tolerations DaemonSet pods get.

signals, default thresholds, reclaim order, no PDBs.

and Resource Management for Pods and Containers: Allocatable, scheduling by requests.

kubectl cordon, kubectl taint and kubectl top node.

and Troubleshooting Clusters: metrics-server, describe node output.

scale-up and scale-down rules, events, the status ConfigMap, annotations.

  • The FailedScheduling reason texts are from the Kubernetes 1.37 scheduler source

(tainttoleration plugin), because the docs show only an older example.

del mismo repositorio

Más Skills

Todos los Skills
k8s-servant
Comunidad

k8s-api-troubleshooting

Use when the Kubernetes API server itself refuses or fails a request, before any controller acts on it: "failed calling webhook", "admission webhook ... denied the request", "ValidatingAdmissionPolicy ... denied request", a ReplicaSet with FailedCreate and no pods, an APIService "False (ServiceNotFound)", "FailedDiscoveryCheck" or "MissingEndpoints", "unable to retrieve the complete list of server APIs", a namespace stuck in Terminating, "no matches for kind" or "ensure CRDs are installed first"

Instalaciones / 1h
1
GitHub Stars
38
Actualizado
20 nov
k8s-servant
Comunidad

k8s-argocd-troubleshooting

Use when Argo CD does not deploy or update an application in Kubernetes: an Application that is OutOfSync, Unknown, Degraded, Missing or Progressing, a ComparisonError ("app path does not exist", "unable to resolve ... to a commit SHA"), an InvalidSpecError ("do not match any of the allowed destinations in project"), a sync that fails ("one or more objects failed to apply"), auto-sync that never runs, resources that stay OutOfSync after a sync, or ApplicationSets that generate nothing. Investiga

Instalaciones / 1h
1
GitHub Stars
38
Actualizado
20 nov
k8s-servant
Comunidad

k8s-cert-manager-troubleshooting

Use when cert-manager does not issue or renew a certificate in Kubernetes: a Certificate that stays READY False, a CertificateRequest that is Pending, Denied or Failed, an Issuer or ClusterIssuer that is not Ready ("Error getting keypair for CA issuer"), "Referenced \"Issuer\" not found", ACME Orders or Challenges stuck pending (HTTP-01, DNS-01, rate limits), a certificate that expired or did not renew, or "failed calling webhook \"webhook.cert-manager.io\"" when applying resources. Investigates

Instalaciones / 1h
1
GitHub Stars
38
Actualizado
20 nov
k8s-servant
Comunidad

k8s-cilium-troubleshooting

Use when networking fails on a Kubernetes cluster that runs Cilium as its CNI: traffic between pods dropped or timing out after a CiliumNetworkPolicy or CiliumClusterwideNetworkPolicy, "Policy denied" drops in Hubble, DNS that stops resolving after an egress policy, HTTP 403 "Access denied" from an L7 rule, toFQDNs rules that do not match, pods stuck ContainerCreating with "unable to connect to Cilium agent", Cilium agent or operator pods not ready, CiliumEndpoints not ready, or IPAM exhaustion.

Instalaciones / 1h
1
GitHub Stars
38
Actualizado
20 nov