scrape

You are orchestrating the full web scraping workflow, from a user's prompt to a working Scrapy spider with web-poet page objects.

Prerequisites

Requires uv. Install if missing.

Input

The raw argument string is $ARGUMENTS. Split it into 2 positional arguments:

1. url: target website URL (first whitespace-separated token)
2. what: what the user wants to extract (the rest after the URL, e.g. "product", "job listing", "recipe" — free text, may contain spaces)

Track progress

Before Stage 1, create exactly these tasks with TaskCreate, in order:

1. "Decide which fields to extract" — /scrape-define
2. "Analyze the website" — /scrape-spec
3. "Create the Scrapy project" — /scrape-ensure-project
4. "Generate the extraction code" — /scrape-codegen (one per data type)
5. "Generate the spider" — /scrape-create-spider

As you launch each skill, TaskUpdate the task to in_progress. Mark it completed only after the skill (all instances of the skill in case of /scrape-codegen) returns successfully. Do not batch updates — flip status at the boundary so the user sees live progress.

Do NOT create tasks inside the sub-skills; they share this session's task list and would duplicate entries.

Stage 1: Define schema

Invoke /scrape-define with the user's arguments. This downloads 1 detail page, discovers fields, and runs a fast terminal approval loop for the schema.

Output: .scrape/{site_name}/ with approved schema (including examples from user-verified values). No stored pages or value files — those come from Stage 2.

Stage 2: Explore and validate

Invoke /scrape-spec .scrape/{site_name}. This downloads more detail and listing pages, compares HTML variants, extracts values, and optionally presents a browser review.

Stage 3: Generate working project

Ensure the Scrapy project

Derive a project name from the domain (e.g., books_toscrape_com).

/scrape-ensure-project ./{project_name} {project_name}

Generate page objects (per data type)

After scrape-spec returns, re-read {site_path}/spec.json to get the final data_types list — scrape-spec may have replaced {data_type} with {data_type}-list when all requested fields were found on list pages.

Identify the primary data type: the first entry in data_types that is not "navigation". If it ends with -list (e.g., "products-list"), the site will use list-page extraction. Otherwise it uses detail-page extraction (the default). Navigation is always present.

Call codegen once per data type in data_types:

/scrape-codegen .scrape/{site_name}/{primary_type} ./{project_name}
/scrape-codegen .scrape/{site_name}/navigation ./{project_name}

Where {primary_type} is whatever scrape-spec wrote — products-list for the list path, products (or the site's data type name) for the detail path. Each call adds its own item class, page object, and fixtures to the project.

Spider generation

After codegen, determine the PO class import paths from the generated files, then:

/scrape-create-spider ./{project_name} {item_po_import_path} {nav_po_import_path}

Also provide start URLs in the prompt (the site URL from spec.json).

When the primary data type ends with -list, the generated item PO class name ends with ListPage (e.g., ProductListPage). scrape-create-spider detects this suffix and automatically generates a list-extraction spider (items yielded directly from parse(), no parse_item callback).

When the primary data type is a plain detail type, the spider uses the standard detail-extraction pattern (navigation in parse(), items in parse_item()).

Tests with a limited crawl.

Report

Created scraping solution for {domain}:
  Project: ./{project_name}/
  Spider: uv run scrapy crawl {spider_name}
  Tests: uv run pytest fixtures/

Offer to help the user deploy to Scrapy Cloud if they wish. It's useful for scheduled or long-running crawls, to keep a job history with results and logs, for job monitoring (with an API that an LLM can use), and more. There is also a free tier.

If they deploy and run it there (via /scrape-scrapy-cloud), the run will by default wait for the job to finish and validate its results (errors, field coverage, and whether items match the request) before reporting back — so expect it to take as long as the crawl does unless they opt out.