k8s-servant/k8s-skills

k8s-fluent-bit-troubleshooting

Use when Fluent Bit on Kubernetes does not deliver logs: the DaemonSet crashes on a config error ("tried to instance a plugin name that doesn't exist", "unknown configuration property", "configuration file contains errors"), an output fails ("failed to flus…

Quelltext ansehen

Verwendung

  1. Installationsbefehl kopieren und das Quellpaket vor der Installation in der eigenen Agentenumgebung prüfen.
  2. Benötigte Werkzeuge, Zugangsdaten und Hosts im Original prüfen.
  3. Eingabe und gewünschtes Ergebnis beschreiben und die Ausgabe mit den Quellen abgleichen.
npx skills add https://github.com/k8s-servant/k8s-skills --skill k8s-fluent-bit-troubleshooting

Vor dem Start

  • Die Installation umfasst weder Modellabo noch API-Kontingent oder Dateizugriffsrechte.
  • Sterne und Installationen zeigen Aktivität, keine unabhängige Qualitäts- oder Sicherheitsprüfung.
Originales Skill-Dokument

Aus dem Quell-Repository gerendert; Überschriften, Beispiele, Code, Tabellen, Links und Bilder bleiben erhalten.

Fluent Bit troubleshooting

Find out why Fluent Bit does not ship logs, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

Fluent Bit runs as a DaemonSet: on every node a tail input reads the container log files under /var/log/containers/, the kubernetes filter adds pod metadata from the API server, and outputs send the records to a backend (Elasticsearch, Loki, an HTTP endpoint, a forwarder). Its own log names each part as [input:tail:tail.0], [filter:kubernetes:kubernetes.0], [output:http:http.0], so the log usually names the cause. The Helm chart keeps the config in a ConfigMap (fluent-bit.conf); the Fluent Operator builds it from custom resources into a Secret.

The commands use placeholders: my-namespace, my-fluent-bit (the DaemonSet, its ConfigMap and its ServiceAccount), my-fluent-bit-pod (one of its pods) and my-backend (the Service an output sends to). Replace them with the real names. Never read Secrets: not the Fluent Operator's rendered config, not Secrets that outputs take credentials from. The ConfigMap may hold credentials inline (HTTP_Passwd, http_api_key, cloud_auth, bearer_token, AWS keys): read it, but never repeat those values. Avoid kubectl describe pod on Fluent Bit: it prints literal environment values. Never request /api/v1/trace/... (a GET turns tracing on) and never POST to /api/v2/reload.

When to use

  • Logs are missing, late, or incomplete at the log backend.
  • Fluent Bit pods crash, restart, or are not Ready.
  • Records arrive without kubernetes fields, or with the raw <time> stdout F <line>

text in log.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

  • The backend refuses what Fluent Bit sends (Loki 400, 401, 429):

k8s-loki-troubleshooting.

  • The backend's Service has no endpoints or a NetworkPolicy blocks it:

k8s-network-troubleshooting.

  • The kubernetes filter is refused by RBAC: k8s-rbac-troubleshooting.
  • Pods are Pending, OOMKilled or evicted: k8s-workload-troubleshooting.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. See the DaemonSet and its pods: ready on every node, restarts.

bash
kubectl get daemonsets -A -l app.kubernetes.io/name=fluent-bit
kubectl get pods -n my-namespace -l app.kubernetes.io/name=fluent-bit -o wide

2. Read Fluent Bit's log, current and before the last restart. Errors and warnings name the input, filter or output.

bash
kubectl logs -n my-namespace daemonset/my-fluent-bit --tail=100
kubectl logs -n my-namespace daemonset/my-fluent-bit --previous --tail=50
kubectl logs -n my-namespace daemonset/my-fluent-bit --tail=500 | grep -E '\[ ?(error|warn)\]'

3. Read the config: inputs (Path, parsers), filters, outputs (Name, Host, Port, Match). Do not repeat credentials you see.

bash
kubectl get daemonset my-fluent-bit -n my-namespace -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}{range .spec.template.spec.volumes[*]}{.name}{" "}{.configMap.name}{.secret.secretName}{.hostPath.path}{"\n"}{end}'
kubectl get configmap my-fluent-bit -n my-namespace -o yaml

4. Check what the outputs and the kubernetes filter depend on: the backend's Service, and the ServiceAccount's access to pods and namespaces.

bash
kubectl get services,endpointslices -A | grep -i my-backend
kubectl get clusterrolebindings -o wide | grep my-fluent-bit
kubectl auth can-i list pods --as=system:serviceaccount:my-namespace:my-fluent-bit

5. Ask Fluent Bit itself (port 2020, if HTTP_Server is on), through the API server's pod proxy, if your credentials allow get on pods/proxy (the built-in view role does not). Each pod reports for its own node.

bash
kubectl get --raw /api/v1/namespaces/my-namespace/pods/my-fluent-bit-pod:2020/proxy/api/v2/health
kubectl get --raw /api/v1/namespaces/my-namespace/pods/my-fluent-bit-pod:2020/proxy/api/v1/metrics

6. Open the matching reference and follow its steps:

What you seeReference
A crash at start, config errors, the Fluent Operator's resources, hot reloadreferences/config-and-operator.md
Outputs failing, retries, dropped chunks, backpressure, missing metadata, tail and parsers, the HTTP endpointsreferences/pipeline.md

Read the results

What the common findings mean (Fluent Bit 5.1):

  • `[config] section 'htttp' tried to instance a plugin name that doesn't exist` then

configuration file contains errors, aborting.: a misspelled plugin Name; the pod exits and goes to CrashLoopBackOff.

  • `[net] getaddrinfo(host='...', err=4): Domain name not found` with

[output:http:http.0] no upstream connections available to ...: the output's Host does not resolve. Then [engine] failed to flush chunk '...', retry in N seconds and, once retries run out, chunk '...' cannot be retried: those logs are dropped. The default Retry_Limit is 1; the Helm chart sets False (retry forever).

  • `[net] TCP connection failed: host:port (Connection refused)`: the name resolves,

but nothing listens there (the backend is down, or the port is wrong).

  • `[filter:kubernetes:kubernetes.0] could not get meta for POD ...`: the filter cannot

read pod metadata (RBAC, Kube_URL, the API server unreachable). Fluent Bit keeps running and ships records without kubernetes fields; the HTTP status is only logged at debug level.

  • Records with `log` holding `2026-... stdout F <line>`: the parser does not match

the runtime's log format; containerd and CRI-O write CRI lines, so use multiline.parser docker, cri.

  • `[input] tail.0 paused (mem buf overlimit`: the output cannot keep up, and the

tail input stops reading until memory frees; lines can be lost on rotation.

State the cause in one sentence, then the evidence you saw (the log lines with their plugin prefix, the config entries), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Fluent Bit's config and RBAC serve every node and every workload's logs: never change them yourself; hand them to the user as proposed fixes. Prefer fixing the source (the Helm values or the Fluent Operator's resources). Fluent Bit reads its config only at start unless hot reload is on, so a changed ConfigMap needs a restart. Write each undo with the values you found.

Fix the config in its source, apply it, and restart the DaemonSet so it reads it:

Proposed fix: review before running. The agent does not run this.
bash
kubectl apply -f my-fluent-bit-configmap.yaml
kubectl rollout restart daemonset/my-fluent-bit -n my-namespace
# Undo: kubectl apply -f my-fluent-bit-configmap-before.yaml, then restart the DaemonSet again

Bind the role the kubernetes filter needs (read on pods and namespaces) to Fluent Bit's ServiceAccount:

Proposed fix: review before running. The agent does not run this.
bash
kubectl create clusterrolebinding my-fluent-bit --clusterrole=my-fluent-bit --serviceaccount=my-namespace:my-fluent-bit
# Undo: kubectl delete clusterrolebinding my-fluent-bit

Once the config or the access is fixed, restart Fluent Bit so it picks it up:

Ask first: the agent runs this only after the user approves it.
bash
kubectl rollout restart daemonset/my-fluent-bit -n my-namespace
# Undo: none needed; the restart only recreates the pods with the same template

Sources

Checked against the Fluent Bit 5.1 documentation and source (v5.1.5), the Helm chart fluent-bit 0.58.3 and the Fluent Operator v3.10.1:

and the Kubernetes filter, including its troubleshooting section.

HTTP output, scheduling and retries, backpressure, monitoring and hot reload.

and the Fluent Operator.

  • Messages from the source:

flb_config.c, flb_engine.c and kube_meta.c. The log lines quoted here were seen on a Kubernetes 1.37 kind cluster with Fluent Bit 5.1.5.

aus demselben Repository

Weitere Skills

Alle Skills
k8s-servant
Community

k8s-api-troubleshooting

Use when the Kubernetes API server itself refuses or fails a request, before any controller acts on it: "failed calling webhook", "admission webhook ... denied the request", "ValidatingAdmissionPolicy ... denied request", a ReplicaSet with FailedCreate and no pods, an APIService "False (ServiceNotFound)", "FailedDiscoveryCheck" or "MissingEndpoints", "unable to retrieve the complete list of server APIs", a namespace stuck in Terminating, "no matches for kind" or "ensure CRDs are installed first"

Installationen / 1h
1
GitHub Stars
38
Aktualisiert
20. Nov.
k8s-servant
Community

k8s-argocd-troubleshooting

Use when Argo CD does not deploy or update an application in Kubernetes: an Application that is OutOfSync, Unknown, Degraded, Missing or Progressing, a ComparisonError ("app path does not exist", "unable to resolve ... to a commit SHA"), an InvalidSpecError ("do not match any of the allowed destinations in project"), a sync that fails ("one or more objects failed to apply"), auto-sync that never runs, resources that stay OutOfSync after a sync, or ApplicationSets that generate nothing. Investiga

Installationen / 1h
1
GitHub Stars
38
Aktualisiert
20. Nov.
k8s-servant
Community

k8s-cert-manager-troubleshooting

Use when cert-manager does not issue or renew a certificate in Kubernetes: a Certificate that stays READY False, a CertificateRequest that is Pending, Denied or Failed, an Issuer or ClusterIssuer that is not Ready ("Error getting keypair for CA issuer"), "Referenced \"Issuer\" not found", ACME Orders or Challenges stuck pending (HTTP-01, DNS-01, rate limits), a certificate that expired or did not renew, or "failed calling webhook \"webhook.cert-manager.io\"" when applying resources. Investigates

Installationen / 1h
1
GitHub Stars
38
Aktualisiert
20. Nov.
k8s-servant
Community

k8s-cilium-troubleshooting

Use when networking fails on a Kubernetes cluster that runs Cilium as its CNI: traffic between pods dropped or timing out after a CiliumNetworkPolicy or CiliumClusterwideNetworkPolicy, "Policy denied" drops in Hubble, DNS that stops resolving after an egress policy, HTTP 403 "Access denied" from an L7 rule, toFQDNs rules that do not match, pods stuck ContainerCreating with "unable to connect to Cilium agent", Cilium agent or operator pods not ready, CiliumEndpoints not ready, or IPAM exhaustion.

Installationen / 1h
1
GitHub Stars
38
Aktualisiert
20. Nov.