k8s-servant/k8s-skills

k8s-rbac-troubleshooting

Use when a Kubernetes request is refused by RBAC: "Error from server (Forbidden)", "User ...

查看源码

使用方法

  1. 复制安装命令,在自己的智能体环境安装前先检查源码包。
  2. 从原始文档核对所需工具、凭据和支持的宿主。
  3. 说明输入材料与期望交付物,并对照来源检查输出。
npx skills add https://github.com/k8s-servant/k8s-skills --skill k8s-rbac-troubleshooting

开始前需了解

  • 安装不包含模型订阅、API 额度或访问文件的权限。
  • 仓库星数和安装量反映活跃情况,不等于独立质量或安全测试。
仓库原始内容

按源仓库内容呈现,保留标题、案例、代码、表格、链接以及原文引用的演示图片。

RBAC troubleshooting

Find out why Kubernetes refuses a request, and propose the smallest grant that fixes it. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

Every request is made by a subject: a user, a group, or a ServiceAccount, which authenticates as system:serviceaccount:<namespace>:<name>. RBAC allows a request only when a binding gives the subject a role whose rules cover the verb, the API group, the resource and the namespace. There are no deny rules: a 403 means that no binding allowed it. A RoleBinding grants its role in its own namespace only; a ClusterRoleBinding grants a ClusterRole everywhere, including cluster-scoped resources such as nodes and namespaces.

The commands use placeholders: my-namespace, my-app (the ServiceAccount and its workload), my-binding and my-role. Replace them with the real names. The built-in view role cannot read Roles, ClusterRoles or their bindings; if they are forbidden, say so and ask the user for them, or work from the error message alone. Never read Secrets, and never read a ServiceAccount token, in a pod or anywhere else.

When to use

  • A log or an error says Forbidden, cannot <verb> resource, or 403.
  • A pod, Job or controller crashes or retries right after a call to the API server.
  • A person or a CI pipeline cannot do what they expect with kubectl.
  • A RoleBinding was created but changes nothing.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • The 403 message has no User "..." cannot part and names an admission webhook or a

policy: k8s-kyverno-troubleshooting, k8s-gatekeeper-troubleshooting. (A webhook authorizer's refusal keeps the User "..." cannot text and adds its own reason after it.)

  • violates PodSecurity: k8s-workload-troubleshooting.
  • The workload fails for another reason once access works: k8s-workload-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. Read the exact refusal. It names the subject, the verb, the resource with its API group, and the namespace or "at the cluster scope". Everything else follows from it.

bash
kubectl logs -n my-namespace deployment/my-app --tail=50
kubectl logs -n my-namespace deployment/my-app --previous --tail=50
kubectl get events -n my-namespace --sort-by=.lastTimestamp

2. Check which identity the workload uses. The pod's ServiceAccount (default if none is set) and whether the API credentials are mounted at all.

bash
kubectl get pods -n my-namespace -l app=my-app -o jsonpath='{range .items[*]}{.metadata.name}{" sa="}{.spec.serviceAccountName}{" automount="}{.spec.automountServiceAccountToken}{"\n"}{end}'
kubectl get serviceaccount my-app -n my-namespace -o yaml

3. Ask the API server. auth can-i with --as asks for another subject; it needs the impersonate verb. If it is refused (cannot impersonate), give the command to the user. Without --as it answers for you.

bash
kubectl auth can-i list configmaps -n my-namespace --as=system:serviceaccount:my-namespace:my-app
kubectl auth can-i --list -n my-namespace --as=system:serviceaccount:my-namespace:my-app
kubectl auth whoami

4. Find the bindings that name the subject, and the role each one refers to. -o wide shows the subjects; a ServiceAccount shows as namespace/name.

bash
kubectl get rolebindings -n my-namespace -o wide
kubectl get clusterrolebindings -o wide | grep my-app
kubectl get rolebindings -A -o wide | grep my-app
kubectl describe rolebinding my-binding -n my-namespace

5. Read the role's rules and compare them with the refused request: verb, API group ("" is the core group), resource (with its subresource, such as pods/log), and resource names.

bash
kubectl get role my-role -n my-namespace -o yaml
kubectl describe clusterrole my-role

6. Open the matching reference and follow its steps:

What you seeReference
A binding that applies to nobody, a role that does not grant what it seems to, aggregated roles, escalation and roleRef errorsreferences/bindings-and-roles.md
401 instead of 403, system:anonymous, missing or expired credentials, impersonation, a 403 from admissionreferences/authentication-and-admission.md

Read the results

What the common findings mean (Kubernetes 1.37):

  • **`User "system:serviceaccount:my-namespace:my-app" cannot list resource "configmaps"

in API group "" in the namespace "my-namespace"**: no binding gives that subject list on configmaps there. The verb matters: kubectl get of one object is get, of all objects list, --watch also needs watch. A role with only get` refuses the list.

  • `... at the cluster scope`: the request is for a cluster-scoped resource (nodes,

namespaces, PersistentVolumes, CRDs) or across all namespaces. Only a ClusterRoleBinding grants that; a ClusterRole bound with a RoleBinding grants its rules inside that one namespace only.

  • A binding that names the right role but the wrong subject: a ServiceAccount subject

in another namespace (default/my-app instead of my-namespace/my-app), kind User with a group's name, or a misspelled name match nobody, without any error.

  • A RoleBinding in another namespace: it grants nothing in my-namespace.
  • **The refusal ends with `rbac: RBAC: role.rbac.authorization.k8s.io "..." not

found**: a binding for this subject names a role that does not exist (often a typo in roleRef.name`).

  • `auth can-i` answers `yes` but the request still fails: the request is made by

another identity (check step 2), or a webhook authorizer or admission refuses it (see the second reference).

  • `cannot impersonate-on:...` or `cannot impersonate resource` when you run

auth can-i --as: your own identity may not impersonate; it says nothing about the subject. Give the command to the user.

State the cause in one sentence, then the evidence you saw (the exact refusal, the binding's subjects and role, the role's rules), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Every change to a Role, ClusterRole or binding grants or removes access, for this workload and for anyone else bound to the same role: never run them yourself; hand them to the user as proposed fixes, granting the narrowest verbs, resources and namespace that fix the refusal. Prefer fixing the source (the Helm chart, the manifests in Git). Write each undo with the values you found.

Add the missing verbs to the role, in its source, for example verbs: [get, list, watch] on configmaps:

Proposed fix: review before running. The agent does not run this.
bash
kubectl apply -f my-app-role.yaml
# Undo: kubectl apply -f my-app-role-before.yaml (the role as it was, with verbs: [get])

Bind the subject that the workload really uses (a ServiceAccount needs its own namespace), or bind a ClusterRole for cluster-scoped resources with a ClusterRoleBinding instead of a RoleBinding:

Proposed fix: review before running. The agent does not run this.
bash
kubectl create clusterrolebinding my-app-node-reader --clusterrole=my-app-node-reader --serviceaccount=my-namespace:my-app
# Undo: kubectl delete clusterrolebinding my-app-node-reader

A binding's roleRef cannot be changed (field is immutable); to point it at another role, delete it and create it again, which removes the access for a moment. Save the binding first, so the undo can restore it:

bash
kubectl get rolebinding my-binding -n my-namespace -o yaml
Proposed fix: review before running. The agent does not run this.
bash
kubectl delete rolebinding my-binding -n my-namespace
kubectl create rolebinding my-binding -n my-namespace --role=my-app-config-reader --serviceaccount=my-namespace:my-app
# Undo: kubectl delete rolebinding my-binding -n my-namespace && kubectl apply -f my-binding-before.yaml

Once access is fixed, a pod in CrashLoopBackOff recovers at its next restart. To retry at once:

Ask first: the agent runs this only after the user approves it.
bash
kubectl rollout restart deployment/my-app -n my-namespace
# Undo: none needed; the restart only recreates the pods with the same template

Sources

Checked against the Kubernetes 1.37 documentation and source (v1.37.1):

roles, bindings, roleRef, subjects, default and aggregated roles, escalation.

verbs per request, kubectl auth can-i, authorizer verdicts, admission after authorization.

and User impersonation: ServiceAccount usernames and groups, 401 and anonymous requests, --as.

default permissions, token checks and rotation.

  • Messages from the source:

responsewriters/errors.go (the Forbidden text) and rbac/validation/rule.go (escalation and subject matching). The refusals quoted here were seen on a Kubernetes 1.37 kind cluster.

来自同一仓库

更多 Skills

全部 Skills
k8s-servant
社区

k8s-api-troubleshooting

Use when the Kubernetes API server itself refuses or fails a request, before any controller acts on it: "failed calling webhook", "admission webhook ... denied the request", "ValidatingAdmissionPolicy ... denied request", a ReplicaSet with FailedCreate and no pods, an APIService "False (ServiceNotFound)", "FailedDiscoveryCheck" or "MissingEndpoints", "unable to retrieve the complete list of server APIs", a namespace stuck in Terminating, "no matches for kind" or "ensure CRDs are installed first"

1小时安装量
1
GitHub Stars
38
最近更新
11月20日
k8s-servant
社区

k8s-argocd-troubleshooting

Use when Argo CD does not deploy or update an application in Kubernetes: an Application that is OutOfSync, Unknown, Degraded, Missing or Progressing, a ComparisonError ("app path does not exist", "unable to resolve ... to a commit SHA"), an InvalidSpecError ("do not match any of the allowed destinations in project"), a sync that fails ("one or more objects failed to apply"), auto-sync that never runs, resources that stay OutOfSync after a sync, or ApplicationSets that generate nothing. Investiga

1小时安装量
1
GitHub Stars
38
最近更新
11月20日
k8s-servant
社区

k8s-cert-manager-troubleshooting

Use when cert-manager does not issue or renew a certificate in Kubernetes: a Certificate that stays READY False, a CertificateRequest that is Pending, Denied or Failed, an Issuer or ClusterIssuer that is not Ready ("Error getting keypair for CA issuer"), "Referenced \"Issuer\" not found", ACME Orders or Challenges stuck pending (HTTP-01, DNS-01, rate limits), a certificate that expired or did not renew, or "failed calling webhook \"webhook.cert-manager.io\"" when applying resources. Investigates

1小时安装量
1
GitHub Stars
38
最近更新
11月20日
k8s-servant
社区

k8s-cilium-troubleshooting

Use when networking fails on a Kubernetes cluster that runs Cilium as its CNI: traffic between pods dropped or timing out after a CiliumNetworkPolicy or CiliumClusterwideNetworkPolicy, "Policy denied" drops in Hubble, DNS that stops resolving after an egress policy, HTTP 403 "Access denied" from an L7 rule, toFQDNs rules that do not match, pods stuck ContainerCreating with "unable to connect to Cilium agent", Cilium agent or operator pods not ready, CiliumEndpoints not ready, or IPAM exhaustion.

1小时安装量
1
GitHub Stars
38
最近更新
11月20日