buildkite-pipelines

Buildkite Pipelines

Pipeline YAML is the core of Buildkite CI/CD. This skill covers writing, optimizing, and troubleshooting .buildkite/pipeline.yml — step types, caching, parallelism, annotations, retry, dynamic pipelines, matrix builds, plugins, notifications, artifacts, and concurrency.

Quick Start

Create .buildkite/pipeline.yml in the repository root:

steps:
  - label: ":hammer: Tests"
    command: "npm test"
    artifact_paths: "coverage/**/*"

  - wait

  - label: ":rocket: Deploy"
    command: "scripts/deploy.sh"
    branches: "main"

Set the pipeline's initial command in Buildkite to upload this file:

steps:
  - label: ":pipeline: Upload"
    command: buildkite-agent pipeline upload

The agent reads .buildkite/pipeline.yml and uploads the steps to Buildkite for execution.

Buildkite looks for .buildkite/pipeline.yml by default. Override the path with buildkite-agent pipeline upload path/to/other.yml.

For creating pipelines programmatically, see the buildkite-api skill. For agent and queue setup, see the queues documentation.

Step Types

For detailed attributes and advanced examples of each step type, see references/step-types-reference.md.

Caching

Caching dependencies is the single highest-impact optimization. Use the cache plugin with manifest-based invalidation:

steps:
  - label: ":nodejs: Test"
    command: "npm ci && npm test"
    plugins:
      - cache#v1.8.1:
          paths:
            - "node_modules/"
          manifest: "package-lock.json"

The cache key derives from the manifest file hash. When package-lock.json changes, the cache rebuilds.

Hosted agents also support a built-in cache key (no plugin needed):

steps:
  - label: ":nodejs: Test"
    command: "npm ci && npm test"
    cache:
      paths:
        - "node_modules/"
      key: "v1-deps-{{ checksum 'package-lock.json' }}"

For hosted agent setup and instance shapes, see the Buildkite hosted agents documentation.

Fast-Fail and Non-Blocking Steps

Cancel remaining jobs immediately when any job fails:

steps:
  - label: ":rspec: Tests"
    command: "bundle exec rspec"
    cancel_on_build_failing: true

Use soft_fail for steps that should not block the build (security scans, linting, coverage):

steps:
  - label: ":shield: Security Scan"
    command: "scripts/security-scan.sh"
    soft_fail:
      - exit_status: 1

A soft-failed step shows as a warning in the UI but does not fail the build. Combine with continue_on_failure: true on a wait step to let downstream steps run regardless.

Parallelism and Dependencies

Parallel execution

Steps at the same level run in parallel by default. Use parallelism to fan out a single step:

steps:
  - label: ":rspec: Tests %n"
    command: "bundle exec rspec"
    parallelism: 10

This creates 10 parallel jobs. Each receives BUILDKITE_PARALLEL_JOB (0-9) and BUILDKITE_PARALLEL_JOB_COUNT (10) as environment variables for splitting work.

For intelligent test splitting based on timing data, see the buildkite-test-engine skill.

Explicit dependencies

Use depends_on to express step-level dependencies without wait:

steps:
  - label: "Build"
    key: "build"
    command: "make build"

  - label: "Unit Tests"
    depends_on: "build"
    command: "make test-unit"

  - label: "Integration Tests"
    depends_on: "build"
    command: "make test-integration"

Unit and integration tests run in parallel after build completes — no wait step needed.

Annotations

Surface build results directly on the build page using buildkite-agent annotate. Supports Markdown and HTML.

steps:
  - label: ":test_tube: Tests"
    command: |
      if ! make test 2>&1 | tee test-output.txt; then
        buildkite-agent annotate --style "error" --context "test-failures" < test-output.txt
        exit 1
      fi
      buildkite-agent annotate "All tests passed :white_check_mark:" --style "success" --context "test-results"

Link to uploaded artifacts in annotations:

- command: |
    buildkite-agent artifact upload "coverage/*"
    buildkite-agent annotate --style "info" 'Coverage: <a href="artifact://coverage/index.html">view report</a>'

Retry

Automatic retry

Retry transient failures by exit status:

steps:
  - label: ":hammer: Build"
    command: "make build"
    retry:
      automatic:
        - exit_status: -1    # Agent lost
          limit: 2
        - exit_status: 143   # SIGTERM (spot instance termination)
          limit: 2
        - exit_status: 255   # Timeout or SSH failure
          limit: 2
        - exit_status: "*"   # Any non-zero exit
          limit: 1

Manual retry

Control whether manual retries are allowed:

retry:
  manual:
    allowed: false
    reason: "Deployment steps cannot be retried"

For comprehensive exit code tables and retry strategy recommendations, see references/retry-and-error-codes.md.

Dynamic Pipelines

Generate pipeline steps at runtime based on repository state. Upload generated YAML with buildkite-agent pipeline upload:

steps:
  - label: ":pipeline: Generate"
    command: |
      .buildkite/generate-pipeline.sh | buildkite-agent pipeline upload

When to use what

Pipelines exist on a spectrum. Pick the simplest option that does the job:

Don't reach for dynamic pipelines for the wrong job

Dynamic generation is the right tool when the steps themselves need to change. For passing data between steps, simpler primitives exist:

• `buildkite-agent meta-data set/get` — small key-value pairs any later step in the same build can read (a version string, a commit SHA, a feature flag).
• Artifacts — files passed between steps (buildkite-agent artifact upload/download).
• Trigger step `env:` — env vars passed to a build in a different pipeline.

If only data needs to move, metadata or artifacts is simpler and safer. See the buildkite-agent-runtime skill.

Bootstrap script

Always start generator scripts with `set -euo pipefail`. Without pipefail, a failing pipeline upload returns the exit code of the last piped command, the build step reports success, and no generated steps appear — the most common dynamic pipeline failure mode.

Example generator that runs tests only for changed services:

#!/bin/bash
set -euo pipefail
CHANGED=$(git diff --name-only HEAD~1)
cat <<YAML
steps:
YAML
for dir in services/*/; do
  svc=$(basename "$dir")
  if echo "$CHANGED" | grep -q "^services/$svc/"; then
    cat <<YAML
  - label: ":test_tube: $svc"
    command: "cd services/$svc && make test"
    key: "test-$svc"
YAML
  fi
done

Set key: on every generated step. It enables depends_on, makes retries idempotent (DuplicateKeyError blocks silent duplication if the upload step re-runs), and gives stable identifiers across builds. Validate locally with buildkite-agent pipeline upload --dry-run before pushing.

Keep uploads under 500 steps per call and 4,000 jobs per build (platform defaults, raisable via support). For larger monorepos, use trigger steps to fan out across separate builds.

For type-checked, unit-testable generators, the Buildkite SDK supports JavaScript/TypeScript, Python, Go, and Ruby. Wrap related steps in group steps once a generator produces more than ~10 steps — adding any group enables DAG mode for the build, and concurrency attributes are rejected on groups (see references/group-steps.md).

A generator step can also read runtime state (meta-data, artifacts, git diff) and upload the next phase of the pipeline — the handler pattern used by multi-stage builds. For this, fan-out/fan-in, and finalizer / always-run steps via pre-exit hooks, see references/dynamic-pipeline-patterns.md. For failure modes, see references/dynamic-pipeline-troubleshooting.md. For advanced generator patterns (Python, monorepo, multi-stage), see references/advanced-patterns.md.

Conditional Execution

Step-level conditions

Use if to conditionally run steps based on build state:

steps:
  - label: ":rocket: Deploy"
    command: "scripts/deploy.sh"
    if: build.branch == "main" && build.message !~ /\[skip deploy\]/

For the full list of condition expressions, see Conditionals.

`[skip ci]` gotcha: Buildkite only checks the HEAD commit message for [skip ci] / [ci skip]. If the tag is in an earlier commit in a multi-commit push, the build still triggers.

Directory-based step filtering (if_changed)

Skip steps when relevant files haven't changed. Only applied by the Buildkite agent when uploading a pipeline. See https://buildkite.com/docs/pipelines/configure/dynamic-pipelines/if-changed.md.

steps:
  - label: ":nodejs: Frontend tests"
    command: "npm test"
    if_changed:
      - "src/frontend/**"
      - "package.json"

For exclude patterns and monorepo configurations, see references/advanced-patterns.md.

For large monorepos, use the Sparse Checkout plugin to check out only .buildkite/ for the upload step — dramatically faster pipeline uploads.

Conditionally running plugins

Step-level if does not prevent plugins from executing. Wrap steps in a group to skip plugins entirely:

steps:
  - group: ":docker: Build"
    if: build.env("DOCKER_PASSWORD") != null
    steps:
      - label: "Build image"
        command: "docker build -t myapp ."
        plugins:
          - docker-login#v2.1.0:
              username: myuser
              password-env: DOCKER_PASSWORD

Matrix Builds

Run the same step across multiple configurations:

steps:
  - label: "Test {{matrix.ruby}} on {{matrix.os}}"
    command: "bundle exec rake test"
    matrix:
      setup:
        ruby:
          - "3.2"
          - "3.3"
        os:
          - "ubuntu"
          - "alpine"
      adjustments:
        - with:
            ruby: "3.2"
            os: "alpine"
          skip: true  # Known incompatible

Valid properties inside each adjustments entry: with, skip, soft_fail, env. The agents: key is not valid inside adjustments — Buildkite rejects the pipeline with "agents is not a valid property on the matrix.adjustments configuration". To route matrix combinations to different queues (e.g., Linux vs Windows agents), use separate steps or a dynamic pipeline generator.

Plugins

Add capabilities with 3-line YAML blocks. Pin versions for reproducibility:

plugins:
  - docker-compose#v5.5.0:
      run: app
      config: docker-compose.ci.yml

Always pin plugin versions (e.g., docker#v5.12.0 not docker#v5). Unpinned versions can break builds when plugins release new major versions.

For private organizational plugins, use full Git URLs — the shorthand syntax only works for public plugins:

plugins:
  - ssh://git@github.com/my-org/my-plugin.git#v1.0.0:
      config: value

Notifications and Artifacts

Add pipeline-level notify: above steps: to send Slack, email, or webhook notifications on build state changes. See Notifications for syntax.

Artifact upload and download

Upload artifacts from steps, download in later steps:

steps:
  - label: "Build"
    command: "make build"
    artifact_paths: "dist/**/*"

  - wait

  - label: "Package"
    command: |
      buildkite-agent artifact download "dist/*" .
      make package

When using artifacts in a Docker build, download artifacts before starting the Docker build since buildkite-agent is not available inside the container:

steps:
  - label: "Docker build"
    command: |
      buildkite-agent artifact download "dist/*" .
      docker build -t myapp .

Concurrency

Limit parallel execution of steps sharing a resource. Always pair concurrency with concurrency_group — without a group name, the limit is silently ignored.

steps:
  - label: ":rocket: Deploy"
    command: "scripts/deploy.sh"
    concurrency: 1
    concurrency_group: "deploy/production"
    concurrency_method: "eager"

Use concurrency_method: "eager" (next available) for independent jobs like deploys. Use the default "ordered" (FIFO) when execution order matters. Set priority (default 0, higher = first) to control which queued jobs run next.

For full concurrency configuration options, see Controlling Concurrency.

For triggering, watching, and debugging pipelines from the terminal, see the buildkite-cli skill.

Common Mistakes

Additional Resources

Reference Files

• `references/step-types-reference.md` — Detailed attribute tables for all step types
• `references/advanced-patterns.md` — Dynamic pipeline generators, matrix adjustments, monorepo patterns, multi-stage pipelines
• `references/retry-and-error-codes.md` — Comprehensive exit code table, retry strategies by failure type
• `references/group-steps.md` — Group step attributes, DAG mode, merging across uploads, no-nesting workaround, job-limit impact
• `references/dynamic-pipeline-troubleshooting.md` — Silent upload failures, quota limits, env var interpolation, duplicate-on-retry, retry storms
• `references/dynamic-pipeline-patterns.md` — Fan-out/fan-in, SDK generation, the handler pattern, finalizer steps, trigger-based fan-out

Examples

• `examples/basic-pipeline.yml` — Minimal working pipeline (test, wait, deploy)
• `examples/optimized-pipeline.yml` — Full-featured pipeline with caching, parallelism, annotations, retry, artifacts, and notifications

For migrating pipelines from other CI systems, see the buildkite-migration skill.

Further Reading

• Buildkite Docs for LLMs
• Defining pipeline steps
• Step types reference
• Pipeline upload
• Conditionals
• Managing pipeline secrets
• Pipeline design best practices