How to Design Open Archives for AI Crawler Load

Open technical archives make a promise: a person can inspect the evidence behind a project, and a machine can retrieve information without asking permission for every page. That promise becomes fragile when a client treats a human-oriented interface as an unlimited extraction API. The recent measurements published by kernel.org are a useful warning, not because public data should be hidden, but because the representation a client chooses can determine who pays for openness.

Kernel.org describes sustained traffic that repeatedly requests rendered commit pages rather than using Git's native transfer mechanism. Its operators report costly HTML rendering across a large, shifting URL space. Those measurements do not identify every client or prove that each request comes from a named AI company. They do show a systems problem familiar to documentation sites, code browsers, public databases and issue trackers: a finite collection can expose an effectively unbounded number of expensive page views.

The goal is not to build a wall around public knowledge. It is to make the low-cost, authorized path obvious and usable, then budget the expensive path so a crawler cannot quietly turn a shared archive into its private compute cluster.

Model the cost of representations, not just requests

A request counter is a weak capacity model when two URLs have radically different costs. A repository clone may transfer already-organized objects that a client can traverse locally. A commit page can require a server to resolve history, invoke a renderer, build navigation and generate a fresh HTML response. Both are public reads, yet they consume different CPU, cache and operator attention.

Start with a route inventory. For each public route family, record the work it triggers: static object delivery, cache lookup, database query, repository walk, full-text search, diff generation or server-side render. Pair that with a few operational measures: median and tail latency, CPU time where available, cache-hit rate, distinct URLs per client, and the ratio of successful responses to useful downstream work. This makes it possible to recognize a client that spreads traffic over unique parameter combinations instead of reusing a bounded set of documents.

Do not assume that a public URL is a stable unit of content. A code browser can expose one commit through several views; a database explorer can produce countless sorting, filtering and pagination combinations. Crawl discovery may see every one as a separate destination even when they resolve to the same underlying record. Identify those equivalence classes before a high-volume client discovers them first.

Give bulk users a first-class, cheaper path

A bulk export hidden in a footer is not an access strategy. If an archive has a native protocol, a snapshot, a documented API, an RSS or Atom feed, or a signed data dump, describe which questions it answers and how often it changes. Put the link near the human interface and return it from machine-readable discovery points where appropriate. The desired behavior should be easier than page-by-page scraping.

For source archives, that can mean cloning or fetching a repository instead of walking commit pages. For public records, it may mean a dated export plus an incremental change feed. For documentation, it may mean a versioned bundle with stable identifiers. These formats also improve reproducibility: a researcher can cite a snapshot and an AI pipeline can retain the exact input it used.

The alternative must remain bounded. Publish clear page sizes, cursor rules, field limits and change semantics. A supposedly efficient endpoint that permits arbitrary joins, unbounded search or every historical revision simply moves the expensive rendering problem behind a different URL. Where a task requires an unusually costly query, make it an explicit job with quotas, a cached artifact, or a human review path.

Keep human pages useful and machine work budgeted

Human-readable pages are still essential. They support inspection, linking, accessibility and ordinary search. Protecting them does not require making them hostile. Cache stable responses, normalize harmless URL variants, set sensible pagination bounds and avoid generating unlimited combinations of filters or comparisons. Use canonical URLs so crawlers and readers converge on the same document rather than multiplying cache keys.

Then set limits that follow cost. A lightweight static page can tolerate a different rate than an on-demand diff or search query. Apply concurrency ceilings and timeouts to the expensive operation itself, not only to the outer HTTP request. Reserve capacity for authenticated collaborators, mirrors and ordinary browsing so that a burst of anonymous work does not consume all available workers. A graceful response such as a cached result, a retry-after signal or a link to the bulk interface is often better than allowing an overloaded renderer to fail unpredictably.

Robots rules can communicate a preference, but RFC 9309 does not make them an authorization control. Treat them as one part of the public contract, alongside published crawler guidance, rate limits and observability. Cooperative clients should identify themselves, use a stable contact channel, follow documented limits and choose the cheapest representation. Evasive clients need technical controls that still work when an identifier is missing or misleading.

Measure whether a defense preserves access

A challenge page or a broad IP block can reduce a graph quickly while also excluding readers, assistive tools, mirrors and legitimate automation. Evaluate defenses with two sets of signals. The first measures abuse: expensive route volume, unique-path churn, renderer saturation and challenge completion. The second measures collateral cost: failed ordinary visits, latency for known useful routes, support reports and access from approved bulk clients.

This is why the kernel.org example matters beyond one service. Its maintainers can distinguish a repository protocol designed for replication from repeated rendering of individual pages. That distinction turns an abstract debate about bots into an operational decision: which access path preserves shared capacity, and what must be constrained when it does not?

Review the answer after every significant interface change. A new search feature, diff view or AI-facing endpoint can create a fresh high-cost surface even if the old archive is well protected. Keep a small runbook that names costly routes, the preferred machine interface, the owner of rate policies and the evidence required before relaxing a limit.

Open archives do not have to choose between universal access and permanent overload. They need an explicit access design: make bulk data portable, make human pages legible, put firm budgets around computation-heavy routes, and measure the experience of legitimate users while doing it. That approach keeps the archive open for the people and tools it was meant to serve.