k8s-servant/k8s-skills

k8s-gatekeeper-troubleshooting

Use when OPA Gatekeeper blocks, warns about or fails to police resources in Kubernetes: "admission webhook \"validation.gatekeeper.sh\" denied the request: [constraint] message", warnings from constraints in warn mode, a constraint that should block but onl…

ソースを見る

使い方

  1. インストールコマンドをコピーし、自分の環境へ導入する前にソースを確認します。
  2. 必要なツール、認証情報、対応環境を原文で確認します。
  3. 入力と成果物を指定し、出力を資料と照合します。
npx skills add https://github.com/k8s-servant/k8s-skills --skill k8s-gatekeeper-troubleshooting

開始前の確認

  • 導入にはモデル契約、API枠、ファイルへのアクセス許可は含まれません。
  • スター数や導入数は活動の指標であり、独立した品質・安全性試験ではありません。
リポジトリの原文

見出し、例、コード、表、リンク、参照画像を含む原文を表示しています。

OPA Gatekeeper troubleshooting

Find out why Gatekeeper denies, warns about or does not police a resource, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

A ConstraintTemplate holds the policy code (Rego, or CEL under K8sNativeValidation) and makes Gatekeeper create a new kind in constraints.gatekeeper.sh (for example K8sRequiredLabels). A constraint of that kind says where the policy applies (spec.match), with which parameters, and what happens on a violation (spec.enforcementAction: deny, dryrun, warn or scoped). The admission webhook checks new requests; the audit checks what already exists and writes the results into each constraint's status.

A constraint is a guardrail someone chose on purpose. The usual fix is to make the resource comply; changing, scoping or exempting a constraint is a decision for its owners.

The commands use placeholders: my-namespace, my-app, k8srequiredlabels (a template) and my-constraint (a constraint of kind K8sRequiredLabels). Replace them with the real names. The built-in view role cannot read Gatekeeper's objects (Gatekeeper adds nothing to it); if the commands are forbidden, say so and ask the user for the constraint and template status. Never read Secrets.

When to use

  • A request is denied by validation.gatekeeper.sh, or prints a Gatekeeper warning.
  • A constraint does not block what it should, or lists violations in its status.
  • Applying a constraint fails because its kind does not exist.
  • A policy that looks at other objects (unique hostnames, namespaces) never fires.
  • Gatekeeper mutations do not change resources.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • The Gatekeeper pods crash or stay Pending: k8s-workload-troubleshooting.
  • The denial comes from Kyverno (validate.kyverno.svc-fail, vpol.validate.kyverno):

k8s-kyverno-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. Read the denial or warning. The text in square brackets is the constraint's name: admission webhook "validation.gatekeeper.sh" denied the request: [<constraint>] <message>. A warning reads Warning: [<constraint>] <message>.

2. List templates and constraints. Every constraint kind is its own resource; constraints lists them all.

bash
kubectl get constrainttemplates
kubectl get constraints

3. Read the constraint: what it matches, its action, its parameters, and the audit's violations.

bash
kubectl get k8srequiredlabels my-constraint -o yaml
kubectl get k8srequiredlabels my-constraint -o jsonpath='{.spec.enforcementAction}{"\n"}{.spec.match}{"\n"}{.spec.parameters}{"\n"}{.status.totalViolations}{"\n"}{.status.violations}{"\n"}'

4. Read the template's status: whether it compiled on every Gatekeeper pod.

bash
kubectl get constrainttemplate k8srequiredlabels -o jsonpath='{.status.created}{"\n"}{range .status.byPod[*]}{.id}{" "}{.errors}{"\n"}{end}'
kubectl get crds | grep constraints.gatekeeper.sh

5. If Gatekeeper itself may be the problem, read its pods, webhooks, Config and logs.

bash
kubectl get deployments,pods -n gatekeeper-system
kubectl get validatingwebhookconfiguration gatekeeper-validating-webhook-configuration -o jsonpath='{range .webhooks[*]}{.name}{" "}{.failurePolicy}{"\n"}{end}'
kubectl get configs.config.gatekeeper.sh -n gatekeeper-system -o yaml
kubectl get namespace my-namespace --show-labels
kubectl logs -n gatekeeper-system deployment/gatekeeper-controller-manager -c manager --tail=100
kubectl logs -n gatekeeper-system deployment/gatekeeper-audit -c manager --tail=100

6. Open the matching reference and follow its steps:

What you seeReference
A denial or warning, violations in a constraint's status, a constraint that does not block (dryrun, warn, scoped, match)references/constraints.md
A template that does not compile, no matches for kind, ValidatingAdmissionPolicies Gatekeeper generatesreferences/templates.md
Gatekeeper stopped enforcing, webhook down, exempt namespaces, sync for referential policies, mutationreferences/webhook-sync-mutation.md

Read the results

What the common findings mean (Gatekeeper 3.23):

  • `denied the request: [<constraint>] <message>`: the constraint has

enforcementAction: deny (the default) and the resource violates it. The message is the template's own text.

  • Violations in `status.violations` but nothing blocked: the constraint is

dryrun (audit only) or warn, or the resource existed before the constraint (the webhook checks only new requests; the audit finds the rest). status.violations holds at most 20 entries by default; totalViolations is the full count.

  • **A template with byPod[].errors such as ingest_error `Could not ingest Rego: ...

regoparseerror`**: the policy code does not compile. Gatekeeper accepts the template object but never creates its constraint kind.

  • `no matches for kind "<Kind>" in version "constraints.gatekeeper.sh/v1beta1"` or

the server could not find the requested resource when applying a constraint: the kind does not exist yet, because its template is missing or failed to compile.

  • Nothing enforced, yet the constraint is right: the Gatekeeper webhook pods are

down. validation.gatekeeper.sh uses failurePolicy: Ignore, so the API server admits everything while Gatekeeper is unreachable, and only the audit notices.

  • A namespace never checked: it carries the label admission.gatekeeper.sh/ignore,

is listed in the Config's excludedNamespaces, or the constraint's match leaves it out. gatekeeper-system is always exempt.

State the cause in one sentence, then the evidence you saw (the denial or violation, the constraint and its action, the template status), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Make the resource comply in its source (manifest, Helm values, GitOps repository). Changing a constraint's action or match, fixing a template, exempting a namespace and editing Gatekeeper's Config or webhooks change what is enforced for everyone: never run them yourself; hand them to the constraint's owners as proposed fixes. Write each undo with the values you found.

Bring back Gatekeeper's webhook after it was scaled to zero (write the undo with the replicas you found):

Ask first: the agent runs this only after the user approves it.
bash
kubectl scale deployment/gatekeeper-controller-manager -n gatekeeper-system --replicas=3
# Undo: kubectl scale deployment/gatekeeper-controller-manager -n gatekeeper-system --replicas=0

Resources admitted while the webhook was down stay as they are; the audit lists them in the constraints' status.violations, and they are fixed in their source.

Sources

Checked against the Gatekeeper 3.23 documentation and source (v3.23.1):

ConstraintTemplates and handling violations: enforcement actions, the denial and warning format.

debugging.

replicating data, mutation, failing closed and emergency recovery.

policy.go and constants.go. The messages quoted here were also seen on a Kubernetes 1.37 kind cluster with Gatekeeper 3.23.1.

同じリポジトリから

関連する Skills

すべての Skills
k8s-servant
コミュニティ

k8s-api-troubleshooting

Use when the Kubernetes API server itself refuses or fails a request, before any controller acts on it: "failed calling webhook", "admission webhook ... denied the request", "ValidatingAdmissionPolicy ... denied request", a ReplicaSet with FailedCreate and no pods, an APIService "False (ServiceNotFound)", "FailedDiscoveryCheck" or "MissingEndpoints", "unable to retrieve the complete list of server APIs", a namespace stuck in Terminating, "no matches for kind" or "ensure CRDs are installed first"

1時間の導入数
1
GitHub Stars
38
更新日
11月20日
k8s-servant
コミュニティ

k8s-argocd-troubleshooting

Use when Argo CD does not deploy or update an application in Kubernetes: an Application that is OutOfSync, Unknown, Degraded, Missing or Progressing, a ComparisonError ("app path does not exist", "unable to resolve ... to a commit SHA"), an InvalidSpecError ("do not match any of the allowed destinations in project"), a sync that fails ("one or more objects failed to apply"), auto-sync that never runs, resources that stay OutOfSync after a sync, or ApplicationSets that generate nothing. Investiga

1時間の導入数
1
GitHub Stars
38
更新日
11月20日
k8s-servant
コミュニティ

k8s-cert-manager-troubleshooting

Use when cert-manager does not issue or renew a certificate in Kubernetes: a Certificate that stays READY False, a CertificateRequest that is Pending, Denied or Failed, an Issuer or ClusterIssuer that is not Ready ("Error getting keypair for CA issuer"), "Referenced \"Issuer\" not found", ACME Orders or Challenges stuck pending (HTTP-01, DNS-01, rate limits), a certificate that expired or did not renew, or "failed calling webhook \"webhook.cert-manager.io\"" when applying resources. Investigates

1時間の導入数
1
GitHub Stars
38
更新日
11月20日
k8s-servant
コミュニティ

k8s-cilium-troubleshooting

Use when networking fails on a Kubernetes cluster that runs Cilium as its CNI: traffic between pods dropped or timing out after a CiliumNetworkPolicy or CiliumClusterwideNetworkPolicy, "Policy denied" drops in Hubble, DNS that stops resolving after an egress policy, HTTP 403 "Access denied" from an L7 rule, toFQDNs rules that do not match, pods stuck ContainerCreating with "unable to connect to Cilium agent", Cilium agent or operator pods not ready, CiliumEndpoints not ready, or IPAM exhaustion.

1時間の導入数
1
GitHub Stars
38
更新日
11月20日