turbo-pipelines

Turbo Pipeline Configuration & Architecture

YAML configuration reference and architecture guide for Turbo pipelines. For interactive pipeline building, use /turbo-builder. For troubleshooting, use /turbo-doctor. For transform implementation, use /turbo-transforms.

Decoded contract events are transform output, not a consumable `<chain>.decoded_logs` dataset. Source EVM contract logs from <chain>.raw_logs, then decode them in a transform (see /turbo-transforms). Pre-decoded token datasets such as <chain>.erc20_transfers, <chain>.erc721_transfers, and <chain>.erc1155_transfers are separate, available datasets; use /datasets to verify chain coverage.

CRITICAL: Always validate YAML with goldsky turbo validate <file.yaml> before showing complete pipeline YAML to the user or deploying.

Quick Start

name: my-first-pipeline
resource_size: s
sources:
  transfers:
    type: dataset
    dataset_name: base.erc20_transfers
    version: 1.2.0
    start_at: latest
transforms: {}
sinks:
  output:
    type: blackhole
    from: transfers

goldsky turbo validate pipeline.yaml   # Validate first
goldsky turbo apply pipeline.yaml -i   # Deploy + inspect

Prerequisites

• Goldsky CLI — if goldsky is not on PATH, follow /auth-setup
• Turbo extension — follow /auth-setup and run its bundled installer with turbo. Do not trigger the interactive auto-installer. Verify goldsky turbo --version succeeds; the published Linux binary requires x64 and glibc 2.39+, and the Mac binary requires Apple Silicon. Windows uses the WSL entry point. An unsupported platform is not a successful installation.
• Shell PATH — restore export PATH="$HOME/.local/bin:$HOME/.goldsky/bin:$PATH" in every new shell/tool call; exports do not persist across calls.
• Logged in — follow /auth-setup and run goldsky login yourself when goldsky project list is not authenticated
• Secrets created for sinks if using PostgreSQL, ClickHouse, Kafka, etc. (see /secrets)

Architecture Decisions

Source Type Selection

Note: Kafka sources are used in production but are not documented in official Goldsky docs. Contact Goldsky support for topic names.

Data Flow Patterns

For detailed pattern diagrams, YAML examples, and multi-chain deployment guidance, read references/architecture-patterns.md.

Resource Sizing

Start small and scale up — defensive sizing avoids wasted resources.

`resource_size` sizes the pipeline, not the sink — they are independent. An l/xl pipeline pointed at an undersized destination produces the "deployed but writing nothing" shape: it validates, deploys, reports Running, and then fails every write. A Neon free tier (512 MB) will not hold a multi-month backfill of a high-volume chain — it errors could not extend file because project size limit (512 MB) has been exceeded and checkpoints time out. Check the destination's storage ceiling against the backfill scope (the start position plus any end bound) before deploying, not after.

Sink Selection

For full sink field specifications, see Sinks in the docs (one page per sink type).

Streaming vs Job Mode

Job mode rules: Runs to completion, auto-deletes ~1hr after finishing. Must delete before redeploying. Cannot pause/resume/restart.

Pipeline Splitting

Default to one pipeline with multiple sinks when the user wants the same source data sent to multiple destinations. A single Turbo pipeline supports multiple sinks natively — each sink has a from: field pointing at a source or transform by name, and sinks run independently (one failing does not block the others, and each can have its own batch_size / batch_flush_interval).

Do NOT split into separate pipelines just because there are multiple destinations. Generating one pipeline per sink duplicates the source ingestion, wastes resources, and decouples deployments that should be atomic.

Split into multiple pipelines only when: sources are fundamentally different (different chains with independent lifecycles), the destinations need different resource sizes, or the user explicitly asks for separate pipelines.

See references/architecture-patterns.md for fan-out (one source → multiple sinks) and fan-in (UNION ALL) YAML examples, and templates/multi-sink-pipeline.yaml / templates/fan-out-pipeline.yaml / templates/fan-in-pipeline.yaml for ready-to-adapt configs.

Configuration Reference

Pipeline Structure

name: my-pipeline          # Required: unique identifier (lowercase, hyphens)
resource_size: s            # Required: xs/s/m/l/xl/xxl
job: true                   # Optional: one-time batch (default: false = streaming)

sources:
  source_name:
    type: dataset           # or: kafka
    # ... source config

transforms:                 # Optional
  transform_name:
    type: sql               # or: script, handler, dynamic_table
    # ... transform config

sinks:
  sink_name:
    type: postgres           # or: clickhouse, kafka, webhook, s3_sink, etc.
    # ... sink config

Source Configuration

Dataset Source

sources:
  my_source:
    type: dataset
    dataset_name: <chain>.<dataset_type>
    version: <version>
    start_at: latest | earliest    # EVM, NEAR, Bitcoin, Stellar
    # start_block: <slot_number>   # Solana only — omit to start at the latest slot
    # end_block: <block_number>    # Solana only: bounded backfill (ignored on EVM)
    # filter: >-                   # Optional: SQL WHERE for source-level pre-filtering
    #   address = '0x...' AND block_number >= 10000000

Use filter for contract addresses and block ranges (coarse pre-filtering). Use transform WHERE for fine-grained filtering.

Set the start position explicitly on every dataset source, and say which one you chose. On the start_at chains (EVM, NEAR, Bitcoin, Stellar) omitting it does not mean "start now": the backend starts from the earliest available data, so the pipeline backfills the full chain history — days of replay and millions of rows before it reaches live data, and the sink has to hold all of it. A block number is not a valid start_at value; bound an EVM range with a block_number predicate in filter (pre-applied at the source), plus job: true for a one-shot backfill — end_block is silently ignored on EVM. Solana is the opposite default: it uses the numeric start_block, and omitting that starts at the latest slot, so a Solana backfill is always something you asked for — bound it with end_block or block_ranges.

For chain prefixes and dataset types, see /datasets.

Kafka Source

sources:
  my_source:
    type: kafka
    topic: base.raw.latest_balances_v2

No start_at or version fields. Optional: filter, include_metadata, starting_offsets.

Transform Configuration

SQL Transform

transforms:
  filtered:
    type: sql
    primary_key: id
    sql: |
      SELECT id, sender, recipient, amount
      FROM source_name
      WHERE amount > 1000

Transform Chaining

Chain transforms using from:

transforms:
  step1:
    type: sql
    primary_key: id
    sql: SELECT * FROM source WHERE amount > 100
  step2:
    type: sql
    primary_key: id
    from: step1
    sql: SELECT *, 'processed' as status FROM step1

For TypeScript, handler, and dynamic table transforms, see /turbo-transforms.

Throttle Transform

Caps the throughput of a stream by buffering records into batches and emitting each batch on a fixed minimum interval. Throttle does not modify data — every input record passes through unchanged. Use it to stay under rate limits of downstream sinks or external APIs, smooth bursty sources, or pace records into HTTP handlers.

transforms:
  throttled:
    type: throttle
    from: my_source
    max_batch_size: 100
    min_batch_interval: 10s

Effective max throughput ≈ max_batch_size / min_batch_interval records per second. Throttle limits the maximum rate, not the minimum — if upstream is slow, batches will be smaller and arrive less frequently.

Place throttle close to the bottleneck (just before the rate-limited sink or handler) so upstream transforms still process at full speed.

Sink Configuration

Quick examples for common sinks. For full field specs of all sink types, see Sinks in the docs (one page per sink type).

`primary_key` placement depends on the sink type; it is not limited to transforms. A SQL transform's key describes its output, while a sink's key controls destination behavior:

For other sink types, check their own field reference before adding or removing primary_key. Mirror uses different sink schemas; do not copy Turbo sink fields into Mirror configs.

PostgreSQL

sinks:
  output:
    type: postgres
    from: my_transform
    schema: public
    table: my_table
    secret_name: MY_POSTGRES_SECRET
    primary_key: id

ClickHouse

sinks:
  output:
    type: clickhouse
    from: my_transform
    table: my_table
    secret_name: MY_CLICKHOUSE_SECRET
    primary_key: id

Pub/Sub (Turbo-only)

sinks:
  output:
    type: pubsub
    from: my_transform
    topic: my-topic
    secret_name: MY_PUBSUB_SECRET

The secret holds a GCP project id and service-account JSON. Topic must already exist in GCP.

Blackhole (Testing)

sinks:
  output:
    type: blackhole
    from: my_transform

Checkpoint Behavior

• Preserved by default when updating a pipeline
• Tied to source names — renaming a source resets its checkpoint
• Tied to pipeline names — renaming the pipeline resets all checkpoints

To reset checkpoints: rename the source or pipeline. Warning: this reprocesses all historical data.

Starter Templates

Template location: templates/ (relative to this skill's directory)

CLI Quick Reference

For lifecycle commands (pause/resume/restart/delete) and monitoring (inspect/logs), see /turbo-operations.

Troubleshooting

See references/troubleshooting.md for CLI hanging, validation errors, and runtime errors.

Related

• `/turbo-builder` — Interactive wizard to build pipelines step-by-step
• `/turbo-doctor` — Diagnose and fix pipeline issues
• `/turbo-operations` — Lifecycle commands and monitoring reference
• `/turbo-transforms` — SQL, TypeScript, and dynamic table transform reference
• `/datasets` — Dataset names and chain prefixes
• `/secrets` — Sink credential management