k8s-servant/k8s-skills

k8s-argocd-troubleshooting

Use when Argo CD does not deploy or update an application in Kubernetes: an Application that is OutOfSync, Unknown, Degraded, Missing or Progressing, a ComparisonError ("app path does not exist", "unable to resolve ...

소스 보기

사용 방법

  1. 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
  2. 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
  3. 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/k8s-servant/k8s-skills --skill k8s-argocd-troubleshooting

시작 전 확인

  • 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
  • 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

Argo CD troubleshooting

Find out why Argo CD does not deploy or update an application, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

An Application says: render this path of this repository at this revision, and apply it to this destination. Argo CD compares the rendered manifests with the cluster and reports two things: sync status (does the cluster match Git: Synced, OutOfSync, Unknown) and health (do the deployed objects work: Healthy, Progressing, Degraded, Missing, Suspended). Conditions on the Application say why it could not compare or sync. An AppProject limits which repositories, destinations and kinds an Application may use.

The commands use placeholders: argocd (where Argo CD and its Applications live, the default), my-app (the Application) and my-namespace (where it deploys). Replace them with the real names.

Never read Secrets. Argo CD keeps repository credentials, cluster credentials and its own settings in Secrets (argocd-secret, and Secrets labelled argocd.argoproj.io/secret-type). Read the ConfigMaps instead (argocd-cm, argocd-rbac-cm, argocd-cmd-params-cm); the Application's conditions say when a credential is missing or wrong.

When to use

  • An Application shows OutOfSync, Unknown, Degraded, Missing or Progressing

for longer than expected.

  • A change merged to Git never reaches the cluster.
  • A sync fails, or resources go OutOfSync again right after a sync.
  • An ApplicationSet creates no Applications, or the wrong ones.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • Argo CD synced, but the deployed pods fail (health Degraded or Progressing because

of crashes, images, probes, Pending): k8s-workload-troubleshooting.

  • The Argo CD components themselves crash or stay Pending:

k8s-workload-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. List the Applications with their sync and health status. The built-in view role cannot read Applications (Argo CD adds nothing to it); if the command is forbidden, say so and ask the user for the Application's status from the Argo CD UI or CLI.

bash
kubectl get applications -n argocd

2. Read the Application: its source, destination, project, sync policy, conditions and the last sync operation.

bash
kubectl describe application my-app -n argocd
kubectl get application my-app -n argocd -o jsonpath='{.spec.source}{"\n"}{.spec.destination}{"\n"}{.spec.project}{"\n"}{.spec.syncPolicy}{"\n"}'
kubectl get application my-app -n argocd -o jsonpath='{.status.conditions}{"\n"}{.status.sync}{"\n"}{.status.health}{"\n"}'
kubectl get application my-app -n argocd -o jsonpath='{.status.operationState.phase}{" "}{.status.operationState.message}{"\n"}'

3. See which resources differ. Each entry of status.resources has a sync status; the failed ones of the last sync have a message.

bash
kubectl get application my-app -n argocd -o jsonpath='{range .status.resources[*]}{.kind}{" "}{.namespace}{"/"}{.name}{" "}{.status}{"\n"}{end}'
kubectl get application my-app -n argocd -o jsonpath='{range .status.operationState.syncResult.resources[*]}{.kind}{" "}{.name}{" "}{.status}{" "}{.message}{"\n"}{end}'

4. Read the AppProject the Application uses: allowed source repositories, destinations and resource kinds.

bash
kubectl get appprojects -n argocd
kubectl get appproject default -n argocd -o jsonpath='{.spec.sourceRepos}{"\n"}{.spec.destinations}{"\n"}{.spec.clusterResourceWhitelist}{"\n"}'

5. If the conditions do not explain it, read the components. The repo-server renders manifests (Git and Helm errors), the application controller compares and syncs.

bash
kubectl get pods -n argocd
kubectl logs -n argocd deployment/argocd-repo-server --tail=100 | grep my-app
kubectl logs -n argocd statefulset/argocd-application-controller --tail=200 | grep my-app
kubectl events -n argocd --for application/my-app

6. Open the matching reference and follow its steps:

What you seeReference
ComparisonError, Unknown sync status: app path does not exist, unable to resolve ... to a commit SHA, Helm or Kustomize errors, repository accessreferences/comparison-errors.md
InvalidSpecError: destinations or repositories not permitted in the project, cluster not foundreferences/projects-and-destinations.md
OutOfSync that never syncs, a failed sync, OutOfSync again after a sync, auto-sync skippedreferences/sync.md
Degraded, Progressing, Missing health; ApplicationSets that generate nothingreferences/health-and-appsets.md

Read the results

What the common findings mean (Argo CD 3.5):

  • Sync `Unknown` with a `ComparisonError` `Failed to load target state: failed to

generate manifest for source 1 of 1: ...: Argo CD could not render the manifests. The text at the end says why: <path>: app path does not exist (wrong path), unable to resolve '<rev>' to a commit SHA (wrong targetRevision`), a Helm or Kustomize error, or repository access.

  • An `InvalidSpecError` `application destination server '<server>' and namespace

'<ns>' do not match any of the allowed destinations in project '<project>'`: the AppProject does not allow that destination. Either the Application targets the wrong namespace or the project is missing a destination.

  • `OutOfSync` and `Missing`, no error, and no sync ever ran: the Application has no

spec.syncPolicy.automated, so Argo CD only reports the difference and waits for someone to sync. This is not a fault, but often not what the user expects.

  • A sync `Failed` with `one or more objects failed to apply, reason: ...`: the API

server refused an object; the per-resource message in syncResult is the API error.

  • `OutOfSync` again right after a successful sync: something in the cluster

changes the objects after Argo CD applies them (a mutating webhook, an operator, an HPA on replicas), or the manifest has defaults the API server fills in differently.

  • Health `Degraded` or `Progressing` with sync Synced: Argo CD did its part; the

deployed workload is not healthy.

State the cause in one sentence, then the evidence you saw (the condition and its message, the spec field, the project rule), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Asking Argo CD to refresh, starting one sync and patching an Application's spec are reversible, so they go under "Ask first" and run only after the user says yes. Changing AppProjects, repository or cluster credentials, syncing with prune or replace, and deleting Applications are never run by you: hand them to the user as proposed fixes. Applications are often themselves managed from Git (app-of-apps, ApplicationSets, or a GitOps repository for Argo CD); a patch on the live Application is then reverted, so propose the change in Git. Write each undo with the values you found.

Point an Application at the right path (to test before fixing Git; the undo puts back the value you found in step 2):

Ask first: the agent runs this only after the user approves it.
bash
kubectl patch application my-app -n argocd --type=merge -p '{"spec": {"source": {"path": "guestbook"}}}'
# Undo: kubectl patch application my-app -n argocd --type=merge -p '{"spec": {"source": {"path": "guestbook-ui"}}}'

Ask Argo CD to compare again now, for example after fixing the repository (the controller removes the annotation when it is done):

Ask first: the agent runs this only after the user approves it.
bash
kubectl annotate application my-app -n argocd argocd.argoproj.io/refresh=normal --overwrite
# Undo: nothing to undo; the annotation only asks for one refresh

Sync once, without prune, when the user wants the current Git revision deployed and the Application has no automated sync (initiatedBy records who asked):

Ask first: the agent runs this only after the user approves it.
bash
kubectl patch application my-app -n argocd --type=merge -p '{"operation": {"initiatedBy": {"username": "my-user"}, "sync": {"revision": "master"}}}'
# Undo: sync the revision of the previous entry in status.history the same way; objects this sync created stay until removed with prune, which is for the user

Turning on automated sync is a change to how the Application behaves from then on; propose it in the source, with the user deciding on prune and selfHeal.

Sources

Checked against the Argo CD 3.5 documentation and source (v3.5.6):

and the FAQ.

and annotations and labels: the operation field and the refresh annotation.

and resource tracking.

automated sync.

upgrade notes for 3.x.

  • Status types and messages from the source:

types.go, path.go, argo.go, appcontroller.go. The messages quoted here were also seen on a Kubernetes 1.37 kind cluster with Argo CD 3.5.6.

같은 저장소의 Skills

더 많은 Skills

모든 Skills
k8s-servant
커뮤니티

k8s-api-troubleshooting

Use when the Kubernetes API server itself refuses or fails a request, before any controller acts on it: "failed calling webhook", "admission webhook ... denied the request", "ValidatingAdmissionPolicy ... denied request", a ReplicaSet with FailedCreate and no pods, an APIService "False (ServiceNotFound)", "FailedDiscoveryCheck" or "MissingEndpoints", "unable to retrieve the complete list of server APIs", a namespace stuck in Terminating, "no matches for kind" or "ensure CRDs are installed first"

1시간 설치 수
1
GitHub Stars
38
업데이트
11월 20일
k8s-servant
커뮤니티

k8s-cert-manager-troubleshooting

Use when cert-manager does not issue or renew a certificate in Kubernetes: a Certificate that stays READY False, a CertificateRequest that is Pending, Denied or Failed, an Issuer or ClusterIssuer that is not Ready ("Error getting keypair for CA issuer"), "Referenced \"Issuer\" not found", ACME Orders or Challenges stuck pending (HTTP-01, DNS-01, rate limits), a certificate that expired or did not renew, or "failed calling webhook \"webhook.cert-manager.io\"" when applying resources. Investigates

1시간 설치 수
1
GitHub Stars
38
업데이트
11월 20일
k8s-servant
커뮤니티

k8s-cilium-troubleshooting

Use when networking fails on a Kubernetes cluster that runs Cilium as its CNI: traffic between pods dropped or timing out after a CiliumNetworkPolicy or CiliumClusterwideNetworkPolicy, "Policy denied" drops in Hubble, DNS that stops resolving after an egress policy, HTTP 403 "Access denied" from an L7 rule, toFQDNs rules that do not match, pods stuck ContainerCreating with "unable to connect to Cilium agent", Cilium agent or operator pods not ready, CiliumEndpoints not ready, or IPAM exhaustion.

1시간 설치 수
1
GitHub Stars
38
업데이트
11월 20일
k8s-servant
커뮤니티

k8s-coredns-troubleshooting

Use when cluster DNS itself misbehaves on Kubernetes: CoreDNS pods in CrashLoopBackOff ("Error during parsing: Unknown directive", "plugin/loop: Loop ... detected"), "reload failed" after a Corefile edit, external names failing while Service names resolve (";; connection timed out; no servers could be reached", SERVFAIL, "plugin/errors: 2 ... i/o timeout"), one domain or stub domain failing, rewrite or hosts entries that do not work, "Plugins not ready: \"kubernetes\"", REFUSED or "concurrent qu

1시간 설치 수
1
GitHub Stars
38
업데이트
11월 20일