octen-extract

Octen Extract

Turn one or more URLs into clean, LLM-ready content. Beyond the page body, each result also carries a category, a page-structure label, and — when you pass a query — query-relevant highlights instead of the full page.

Requires API Key: Get one at https://octen.ai · Set it: export OCTEN_API_KEY=your-api-key

API Key Setup

Before extracting, ensure `OCTEN_API_KEY` is set. On `401`, stop and tell the user a key is required (https://octen.ai), help them configure it, then continue. Configure the key for the relevant agent (same as other Octen skills):

• Claude Code — add { "env": { "OCTEN_API_KEY": "your-key" } } to ~/.claude/settings.json
• Cursor / generic shell — export OCTEN_API_KEY="your-key" in ~/.zshrc or ~/.bashrc
• Codex — ~/.codex/config.toml → [shell_environment_policy], set = { OCTEN_API_KEY = "your-key" }
• OpenClaw — add OCTEN_API_KEY=your-key to ~/.openclaw/.env

Endpoint

POST https://api.octen.ai/extract

Authentication: X-Api-Key: <API_KEY> header · Content-Type: application/json

Quick Start (cURL)

Batch extract (full content)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://example.com", "https://octen.ai"],
    "format": "markdown"
  }'

Query-driven highlights (instead of full content)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://en.wikipedia.org/wiki/Python_(programming_language)"],
    "query": "async programming"
  }'

Hard site (advanced mode)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://www.reddit.com/r/LocalLLaMA/"],
    "mode": "advanced",
    "timeout": 60
  }'

Mixed batch, difficulty unknown (auto mode)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://example.com", "https://octen.ai", "https://news.ycombinator.com/item?id=1"],
    "mode": "auto",
    "timeout": 60
  }'

Page links (outbound references first)

curl -s -X POST "https://api.octen.ai/extract" \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: ${OCTEN_API_KEY}" \
  -d '{
    "urls": ["https://octen.ai"],
    "include_links": {"scope": "prefer_external", "max_links": 50}
  }'

Parameters

| Parameter | Type | Required | Default | Description | |--|--|--|--|--| | urls | string[] | Yes | - | 1–20 URLs to fetch in one batch | | mode | string | No | standard (when omitted) | standard, advanced, or auto — see Choosing a mode | | query | string | No | - | When set, each result returns query-relevant highlights and OMITS full_content (max 500 chars) | | max_age_seconds | integer | No | 86400 | Accept cached results within this age (min 300); older cached versions are re-fetched | | format | string | No | markdown | Content format: markdown or text | | timeout | integer | No | 30 | Per-URL fetch budget in seconds (1–60) | | include_images | boolean | No | false | Include image resources found on the page (also returns cover_image when the page has one) | | include_videos | boolean | No | false | Include video URLs found on the page | | include_audio | boolean | No | false | Include audio URLs found on the page | | include_links | object | No | - | Use it to discover a site's pages or follow outbound references. Returns the page's links: {scope, max_links}. scope: prefer_internal (default; same-site links first) or prefer_external (other registered domains first); page order is kept within each group. max_links: 1–1000 (default 200; out of range → 400, not clamped). {} uses both defaults |

Choosing a mode

You pick the mode; omitting it means standard.

• `standard` (default): fastest and cheapest. Right for ordinary, well-structured pages without anti-bot protection — news, blogs, docs, product pages, static sites. On login-walled, JS-heavy, or anti-bot sites it can return empty content or just a page skeleton while still reporting `status: "success"`.
• `advanced`: highest success rate — renders in a real browser with stronger anti-bot handling; slower and 2.5x the price. Use it for known hard sites: anti-bot/WAF-protected sites, JS-heavy SPAs, social sites (Reddit, X, LinkedIn), academic sites (ResearchGate), dynamically loaded content.
• `auto`: decides per URL — standard where it suffices, advanced only for hard URLs; each URL is billed at the mode it actually used. Use it for a mixed batch when you can't tell which URLs are hard. If most URLs are hard, its cost approaches all-advanced.

| Scenario | mode | |--|--| | News / blogs / docs / product pages (ordinary sites) | standard (or omit) | | Known hard sites (Reddit / X / LinkedIn / ResearchGate / anti-bot / SPA) | advanced | | Mixed batch, unsure which are hard | auto | | Cost-sensitive, mostly ordinary sites | standard | | Quality first, mostly hard sites | advanced |

Rule of thumb: all ordinary sites → standard (or omit); known hard sites → advanced; mixed or unsure → auto.

Recovery: if a standard result failed, or came back empty/skeletal (page_structure.primary is "No Main Content", or very short content on a page that should have a body), retry just those URLs with "mode": "advanced".

Timeout: advanced and auto are slower — raise timeout (e.g. 60).

Response Format

| Field | Type | Description | |--|--|--| | request_id | string | Unique request identifier | | data.results[] | array | One result per URL | | data.results[].url | string | The URL | | data.results[].status | string | success or failed | | data.results[].resolved_mode | string? | standard or advanced — the mode actually used for this URL. Present on success, absent on failure. May differ from the requested mode (auto picks per URL; advanced can resolve to standard) | | data.results[].error_message | string? | Why it failed (when status = failed) | | data.results[].title | string? | Page title | | data.results[].full_content | string? | Cleaned page content (when no query) | | data.results[].highlights | string[]? | Query-relevant snippets (when query is set) | | data.results[].category | object? | {primary, secondary} — what the page is about | | data.results[].page_structure | object? | {primary, secondary} — what kind of page it is | | data.results[].time_published | string? | Publish time, ISO 8601 | | data.results[].time_last_crawled | string? | Last crawl time, ISO 8601 | | data.results[].favicon | string? | Page favicon URL — returned by default when available | | data.results[].cover_image | object? | {url} — page cover image (when include_images is true and the page has one) | | data.results[].images / videos / audio | object[]? | [{url}] — media found on the page (when the matching include_* is true) | | data.results[].links | object[]? | [{url, anchor_text, is_external}] — page links (on success, when include_links is set and links were found) | | meta.usage.total_urls | integer | URLs requested | | meta.usage.successful_urls | integer | URLs successfully fetched | | meta.usage.successful_by_mode | object | {standard_urls, advanced_urls} — successful URLs per mode. Actual billing is authoritative from these counts, not from summing resolved_mode | | meta.warning | string | e.g. "1 URL(s) failed and were not billed"; empty string when there is nothing to report |

meta is top-level (a sibling of data), not data.meta.

Error Codes

| HTTP Status | Description | |--|--| | 400 | Missing or invalid parameter | | 401 | Invalid or missing API key | | 403 | Insufficient balance | | 429 | Rate limited | | 500 | Internal server error |

Notes

• Partial success is first-class: a failed URL is returned with status: "failed" + an error_message; sibling URLs in the same batch still succeed. Always check per-result status.
• Pass query only when you want focused highlights for RAG/grounding; omit it to get the full cleaned page.
• Pricing: standard $1 / 1K successful URLs, advanced $2.5 / 1K. Failed URLs are not billed.
• category and page_structure let you route or filter pages (e.g., skip an index/operation page, keep a content page) without reading the whole body.