gke-workload-identity

GKE Workload Identity Federation Troubleshooting Skill

Use this skill to systematically diagnose why a Pod using Workload Identity Federation for GKE cannot authenticate to Google Cloud APIs. Typical symptoms:

• `HTTP/403 ... Permission 'iam.serviceAccounts.getAccessToken' denied on

resource`

• google.auth.exceptions ... could not find default credentials /

ComputeEngineCredentials cannot find the metadata server

• API calls that unexpectedly use the node's default Compute Engine service

account instead of the workload's identity.

Read-only boundary

This skill is diagnostic and non-interactive. It only reads cluster, IAM, and logging state and proposes fixes as commands or GitOps manifest changes for a human to apply. It must never create or modify IAM bindings, KSA annotations, node pools, or clusters automatically. When evidence is missing or the fix requires a privileged change, summarize findings and hand off to a human (see Step 7).

Output discipline (apply to every conclusion)

When you report a diagnosis, always:

1. Name the single most-likely root cause (not an open-ended list of

possibilities).

1. Explicitly rule out the other plausible causes, citing the evidence that

excludes them. In particular, when the cause is node-pool configuration, state plainly that it is not a KSA annotation or IAM binding problem; when the cause is a missing role on a target resource, state that Workload Identity itself is not misconfigured.

1. Give the exact remediation as a proposed change for a human — a concrete

gcloud command (or GitOps manifest edit) with the real identifiers filled in — and never apply it automatically.

Diagnostic Workflow

Step 0: Context discovery & time window

Collect (from the user or the failing resource): PROJECT_ID, PROJECT_NUMBER, CLUSTER, cluster LOCATION, NAMESPACE, the KSA the Pod runs as, the node and node pool the Pod is scheduled on (used in Step 2), the target resource / API being called, and the exact error string. Define a time window around the first observed failure for log queries.

# Resolve the project number (used in the direct-binding principal identifier).
gcloud projects describe "{PROJECT_ID}" --format="value(projectNumber)"

# Confirm which KSA the workload runs as.
kubectl get pod "{pod_name}" -n "{namespace}" \
  -o jsonpath='{.spec.serviceAccountName}'

# Identify the node the Pod runs on, then the node pool that node belongs to
# (Step 2 checks the node pool's Workload Identity mode).
NODE=$(kubectl get pod "{pod_name}" -n "{namespace}" -o jsonpath='{.spec.nodeName}')
kubectl get node "$NODE" \
  -o jsonpath='{.metadata.labels.cloud\.google\.com/gke-nodepool}'

Step 1: Capture the exact error signature

Read the workload's own logs and the gke-metadata-server logs to classify the failure.

kubectl logs "{pod_name}" -n "{namespace}" --all-containers --prefix
kubectl describe pod "{pod_name}" -n "{namespace}"

Equivalent via Cloud Logging (preferred for historical events). Open it as a Logs Explorer deep link — URL-encode the query and append the project and time window: https://console.cloud.google.com/logs/query;query={URL_ENCODED_QUERY};timeRange={start}%2F{end}?project={project_id} (encode / as %2F, or use ;duration=PT1H for a rolling hour):

resource.type="k8s_container"
resource.labels.namespace_name="{namespace}"
resource.labels.pod_name="{pod_name}"
severity>=WARNING

Classify the signature:

• iam.serviceAccounts.getAccessToken denied / HTTP/403 → the workload is

using the GSA impersonation path. This is expected for the legacy setup, but if you intend to use direct binding, it means the KSA still carries a leftover iam.gke.io/gcp-service-account annotation (or the client SDK is configured to impersonate) and is unintentionally impersonating a GSA; go to Step 3.

• 403 PERMISSION_DENIED on the target API/resource (no getAccessToken

in the error) → the resolved identity lacks the required IAM role on that resource; go to Step 4.

• could not find default credentials / cannot find the metadata server →

metadata-server connectivity or a startup race; go to Step 5.

• Calls succeed but as the node default service account → Workload

Identity is not in effect for this node pool; go to Step 2.

Step 2: Verify Workload Identity is enabled (cluster + node pool)

Both the cluster and the node pool the Pod runs on must have Workload Identity enabled. A node pool with GCE_METADATA (instead of GKE_METADATA) causes Pods to fall back to the node's default Compute Engine service account.

# Cluster must have a workload identity pool (PROJECT_ID.svc.id.goog).
gcloud container clusters describe "{cluster}" --location "{location}" \
  --format="value(workloadIdentityConfig.workloadPool)"

# Node pool must have workloadMetadataConfig.mode = GKE_METADATA.
gcloud container node-pools describe "{node_pool}" --cluster "{cluster}" \
  --location "{location}" \
  --format="value(config.workloadMetadataConfig.mode)"

• Empty workload pool → Workload Identity is not enabled on the cluster.
• Node-pool mode is GCE_METADATA (or empty) → the node pool is not using

the GKE metadata server; this is the usual cause of "runs as the node default service account". Remediation: enable --workload-metadata=GKE_METADATA on the node pool (propose to a human; recreates nodes). This is a node-pool configuration problem — not a KSA annotation or IAM binding problem — so do not change KSA annotations or IAM bindings to fix it. When the symptom is "runs as the node default service account", say so explicitly: the root cause is the node pool's workloadMetadataConfig.mode, and the KSA annotation and IAM bindings are ruled out as the cause. Propose the exact fix, e.g.:

gcloud container node-pools update "{node_pool}" --cluster "{cluster}" \
  --location "{location}" --workload-metadata=GKE_METADATA

Step 3: Verify the KSA → identity binding

There are two supported models. Prefer direct binding (current default); treat GSA impersonation as the legacy path.

How the two models fail differently: with direct binding the KSA principal accesses resources directly, so failures show up as a plain 403 PERMISSION_DENIED on the target API (fix in Step 4). A 403 iam.serviceAccounts.getAccessToken instead means an impersonation attempt — intended under the legacy path, or unintended if a leftover iam.gke.io/gcp-service-account annotation remains on a KSA that was meant to use direct binding.

(a) Direct KSA binding (no GSA impersonation). The KSA principal is granted roles directly. Construct the principal identifier and search for its bindings:

principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME}

(b) Legacy: KSA + GSA impersonation. The KSA must be annotated to point at a GSA, and the KSA must hold roles/iam.workloadIdentityUser on that GSA.

# The KSA annotation must reference the intended GSA.
kubectl get serviceaccount "{ksa_name}" -n "{namespace}" \
  -o jsonpath='{.metadata.annotations.iam\.gke\.io/gcp-service-account}'

# The GSA's IAM policy must bind the KSA member to workloadIdentityUser.
gcloud iam service-accounts get-iam-policy \
  "{gsa_name}@{project_id}.iam.gserviceaccount.com" \
  --format=json
# Expect a binding: role roles/iam.workloadIdentityUser,
# member serviceAccount:{PROJECT_ID}.svc.id.goog[{NAMESPACE}/{KSA_NAME}]

If the annotation is present but the binding is missing, the binding was likely removed — check the setIamPolicy audit logs around the failure time to find the responsible principal.

Step 4: Verify IAM permissions on the target resource

Even with a correct binding, the identity (the KSA principal for direct binding, or the GSA for legacy) must hold the role required by the API call (for example roles/storage.objectViewer). The standard IAM Policy Troubleshooter has limited support for Workload Identity principals; use Cloud Asset Inventory to search all IAM policies for the principal instead.

# Direct binding: search for the KSA principal's bindings across the project.
gcloud asset search-all-iam-policies \
  --scope="projects/{PROJECT_ID}" \
  --query='policy:"{PROJECT_ID}.svc.id.goog"'

# Legacy: search for the GSA's bindings on the target resource's project.
gcloud asset search-all-iam-policies \
  --scope="projects/{TARGET_PROJECT_ID}" \
  --query='policy:"{gsa_name}@{project_id}.iam.gserviceaccount.com"'

If no binding grants the required role on the target resource, that missing role is the root cause (and Workload Identity itself is not misconfigured). Present the fix as a proposed change for a human to apply — the exact add-iam-policy-binding with the real principal and role — never applying it automatically. For example, for direct binding on a project-level resource:

gcloud projects add-iam-policy-binding "{TARGET_PROJECT_ID}" \
  --member="principal://iam.googleapis.com/projects/{PROJECT_NUMBER}/locations/global/workloadIdentityPools/{PROJECT_ID}.svc.id.goog/subject/ns/{NAMESPACE}/sa/{KSA_NAME}" \
  --role="{REQUIRED_ROLE}"   # e.g. roles/storage.objectViewer

Step 5: GKE metadata server & connectivity

Applies when the signature is could not find the metadata server, cannot find the metadata server, or a connection/timeout error. The gke-metadata-server DaemonSet (in kube-system) brokers the token exchange on each node; requests to 169.254.169.254 are redirected to it, so a Pod fails closed if it cannot reach a healthy metadata-server Pod on its node.

(a) Check gke-metadata-server Pod health on the workload's node.

# The DaemonSet Pods must be healthy on the workload's node.
kubectl get pods -n kube-system -l k8s-app=gke-metadata-server -o wide

# A gke-metadata-server Pod can be OOM-evicted when the cluster has many
# (>3,000) Kubernetes service accounts. Look for CrashLoopBackOff, then confirm
# the eviction was OOMKilled.
kubectl get pods -n kube-system | grep CrashLoopBackOff
kubectl describe pod {gke_metadata_server_pod} --namespace=kube-system | grep OOMKilled

(b) Inspect the `gke-metadata-server` logs (historical, via Cloud Logging; open as a Logs Explorer deep link, see Step 1):

resource.type="k8s_container"
resource.labels.namespace_name="kube-system"
labels."k8s-pod/k8s-app"="gke-metadata-server"
severity>=WARNING

(c) Connectivity test from the affected Pod — it must reach the metadata server and (for some client libraries) resolve its DNS name:

# Token endpoint via the hardcoded metadata IP (a healthy path returns a token).
kubectl exec {pod_name} -n {namespace} -- \
  curl -sS -H 'Metadata-Flavor: Google' \
  'http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token'

Possible causes:

• Startup race: the metadata server needs a few seconds after a Pod

starts. Applications that authenticate immediately may fail; the fix is application-side retries or an initContainer that waits for the metadata server, not a cluster change.

• DNS: some client libraries resolve metadata.google.internal; broken

in-cluster DNS surfaces as cannot find the metadata server. As a workaround, set GCE_METADATA_HOST=169.254.169.254 to skip DNS resolution.

• Network policy / egress: a cluster network policy must allow egress to

169.254.169.254/32 on port 80 (GKE Dataplane V2), or 169.254.169.252/32 on port 988. A default-deny egress rule that blocks HTTPS to the public Security Token Service (sts.googleapis.com:443) produces 504 Gateway Timeout or context deadline exceeded.

• `gke-metadata-server` crashing / high restarts: see (a) — reduce the

number of Kubernetes service accounts (<3,000) to restore functionality.

Step 6: "Invalid form of account ID" edge case (legacy GSA impersonation)

Applies only to the legacy GSA impersonation setup (a KSA annotated with iam.gke.io/gcp-service-account), not to direct principal binding.

Signature: an operation that requires an IAM service account email (for example, manually creating a Cloud Storage signed URL) fails with:

ERROR: Invalid form of account ID SERVICEACCOUNT_NAME.svc.id.goog.
Should be [Gaia ID | Email | Unique ID | ] of the account

Cause: for a KSA linked to a GSA via annotation, the GKE metadata server by default returns SERVICEACCOUNT_NAME.svc.id.goog as the identifier, which is not a valid IAM service account email. This is not a missing IAM binding and does not require reverting the setup.

Fix (propose to a human): add the iam.gke.io/return-principal-id-as-email="true" annotation to the Pod's KSA so the metadata server returns the identity in the expected form:

kubectl annotate serviceaccount "{ksa_name}" --namespace "{namespace}" \
  iam.gke.io/return-principal-id-as-email="true"

Step 7: Remediation boundary & escalation

• Present the root cause + evidence (the exact error, the missing binding

/ annotation / node-pool mode, or the metadata-server state). Include a Cloud Logging deep link (see Step 1) to the supporting entries.

• Propose the fix as a command or GitOps manifest change for a human to

apply — never modify IAM, KSAs, or node pools automatically.

Escalate (instead of proposing more self-service diagnostics) when either:

• the relevant logs are unavailable (excluded by a filter, or past the log

bucket's retention); or

• the identity/binding is correct, the target-resource IAM is correct, and the

metadata server is healthy, yet the failure persists (undetermined root cause).

In those cases: state the limitation plainly, summarize the findings gathered (cluster/node-pool config, bindings, annotations, metadata-server state, and any setIamPolicy audit logs), and route to GKE support / engineering escalation. Do not fabricate a diagnosis when evidence is missing.

References

This skill is derived from public Google Cloud documentation:

• Troubleshoot GKE authentication issues

— the Workload Identity Federation section: WI-enabled check, KSA annotation, roles/iam.workloadIdentityUser, iam.serviceAccounts.getAccessToken 403, iam.gke.io/return-principal-id-as-email, metadata-server DNS and the `gke-metadata-server` Pod is crashing guidance.

• About Workload Identity Federation for GKE

— identities, the GKE metadata server, and why Pods don't inherit the node identity.

• Configure applications to use Workload Identity Federation for GKE

— node-pool --workload-metadata=GKE_METADATA, the direct-binding principal://... identifier, and the legacy serviceAccount:PROJECT_ID.svc.id.goog[NAMESPACE/KSA_NAME] member format.

• Search IAM policies (Cloud Asset Inventory)

— gcloud asset search-all-iam-policies for auditing a principal's bindings.

• View GKE logs

— Cloud Logging queries for workload and system logs.