z-k8s-deploy

Kubernetes Deploy

A deployed manifest earns trust the same way a Dockerfile does: nothing runs as root, nothing gets more than it needs, and the cluster can tell a healthy pod from a broken one before traffic hits it.

securityContext

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 65532
    seccompProfile: { type: RuntimeDefault }
  containers:
    - name: app
      securityContext:
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
        capabilities: { drop: ["ALL"] }

The kubelet enforces runAsNonRoot against the effective container user; an explicit nonzero runAsUser can override an image's root default. Admission policies may impose additional checks. With readOnlyRootFilesystem, provide writable mounts only for required scratch space.

Resources

resources:
  requests: { cpu: 100m, memory: 128Mi }
  limits: { memory: 256Mi }

Set a memory limit — an OOM-killed pod restarts cleanly. Think twice before a CPU limit: the kernel's CFS bandwidth controller throttles in fixed periods regardless of how briefly the burst lasted, which shows up as latency spikes worse than just letting the pod use spare node capacity. requests drive scheduling and are the number that actually matters; size them from observed usage, not a guess.

Probes and graceful shutdown

livenessProbe:  { httpGet: { path: /healthz, port: 8080 }, periodSeconds: 10, failureThreshold: 3 }
readinessProbe: { httpGet: { path: /readyz,  port: 8080 }, periodSeconds: 5 }
startupProbe:   { httpGet: { path: /healthz, port: 8080 }, periodSeconds: 2, failureThreshold: 30 }
lifecycle:
  preStop:
    exec: { command: ["sleep", "5"] }

The probes and lifecycle above belong to the container. Set spec.terminationGracePeriodSeconds: 30 on the Pod. The exec hook requires sleep in the image; for distroless images, use a supported lifecycle sleep handler for the cluster version or an application-supported drain hook.

• liveness — restart the container if it's wedged; keep this cheap, no dependency checks.
• readiness — pull the pod out of Service endpoints without restarting it; this is where a broken DB connection belongs.
• startup — gates the other two on slow-booting apps so liveness doesn't kill a container still warming up.

On kubectl delete/rollout, the kubelet fires preStop before sending SIGTERM, and only sends SIGTERM once preStop returns. Endpoint removal from the Service and the SIGTERM race each other across kube-proxy/ingress — a short sleep in preStop buys time for that propagation to catch up so in-flight requests don't land on a pod that's already gone. terminationGracePeriodSeconds must exceed preStop duration plus the app's own shutdown timeout, or the kubelet SIGKILLs mid-drain.

PDB, HPA, NetworkPolicy — when they earn their keep

None of these is a default add for a single-tenant, low-traffic service — each is real cost (another object to reconcile, another failure mode to debug) that should track a real requirement.

Helm chart structure

• values.schema.json alongside values.yaml — helm lint/helm install reject malformed values before they reach the API server.
• templates/_helpers.tpl for shared label/name templates ({{ include "app.labels" . }}) — one definition, not copy-pasted blocks per template.
• Pin Chart.yaml version (chart semver) separately from appVersion (the image tag); bump both on every release.
• Keep templates declarative. A chart that needs {{ if }}/{{ range }} nesting three levels deep to express one deploy usually means the values schema is wrong, not that the template needs another conditional.

Rollout and debug flow

kubectl rollout status deployment/app --timeout=5m
kubectl rollout undo deployment/app                    # back to previous revision
kubectl rollout undo deployment/app --to-revision=N

Triage order on a stuck or crashing pod: kubectl describe pod <pod> first — the Events section at the bottom shows scheduling failures, image pull errors, and probe failures before you ever need a log line. Then kubectl logs <pod> (add --previous for a pod that already restarted). Then kubectl get events --sort-by=.lastTimestamp for cluster-wide context (evictions, OOM kills elsewhere).

For a distroless image with no shell, kubectl debug -it pod/app --image=busybox --target=app attaches an ephemeral debug container sharing the target's process namespace — no restart, no rebuilding the image with a shell just to curl from inside it.

Config and secrets

ConfigMap for non-sensitive config only. Never put secrets in values.yaml or a plain Secret manifest — both land unencrypted in the Helm release history and in version control if committed. Source secrets from an external store (cloud KMS-backed secret manager, External Secrets Operator, Sealed Secrets) that injects them at deploy time; the chart references the resulting Secret name, never the value.

Do not

• Set runAsNonRoot: true and stop there — pair it with a numeric runAsUser or a USER in the image that's actually non-root, or the pod fails to start.
• Add a CPU limit by default — profile first; CFS throttling under a limit often causes worse tail latency than no limit at all.
• Rely on livenessProbe to check downstream dependencies — a flaky DB then restart-loops every pod that depends on it.
• Skip terminationGracePeriodSeconds tuning on a service with real in-flight requests — the 30s default is a guess, not a measurement.
• Put a NetworkPolicy in the chart without confirming the cluster's CNI enforces it — it silently does nothing on Flannel.

Verify

helm lint ./chart
helm template ./chart | kubeconform -summary -
helm template ./chart | kubectl diff -f -
kubectl rollout status deployment/app --timeout=5m