The Crawl Price Index.
Sign in Terminal — €49/mo

Futures

Four ways this could go.

Where the crawl economy ends up by 2030-2033, written down in advance, with the readings from our own census that would prove or disprove each one.

10 published editions since 2026-08-23 scored against edition 2026-09-23 6 scored editions 3 of 4 destinations scored below; the fourth is registered and withheld.

The archive’s row for this edition was scored on bytes the feed has since replaced (dashboard.reachability.states.bot_walled 5038 → 5094; dashboard.reachability.states.alive 43978 → 44127). This page re-scored it from the feed shown; futures-history.json is the census’s to correct and has not been touched.

The machinery, as the census reads it

What the census records at one ranked domain, and in the listing it reads

How a page reaches a machine today, as the census can observe it: who knocks, what is declared, what the wire records — and beneath it, the context the instrument cannot see.

Fig. 1 instrument the census’s figure
The door, as the census reads it — today roles, not organisations · no figures REQUESTS the census (this instrument) knocks three ways: our own identified bot; then, where a door answered, a named AI crawler's user-agent, sent by us, and a browser's user-agent, not a browser; records the answers a reader's browser a reader's agent a lab's training crawler a search crawler an archive crawler these knock too; unobserved DECLARED robots.txt allow / disallow per crawler declared, not enforced OBSERVED the door DNS · a wall, seen when it names itself in a header · an origin — one status comes back; which of them answered is not recorded a registry sellers list a machine-payable price per endpoint reads the declaration knocks declared against, served reads the listing WHAT THE WIRE RECORDS the status is observed; which hop produced it is not 200 served 402 priced a price named in a payment header, or none settled off the wire 403 refused a challenge a header from a wall that names itself 429 slowed no answer read by the census observable at the door not observable at the door The door, as the census reads it — today roles, not organisations · no figures REQUESTS a reader's browser a reader's agent a lab's training crawler a search crawler an archive crawler these knock too; unobserved the census (this instrument) knocks three ways: our own identified bot; then, where a door answered, a named AI crawler's user-agent, sent by us, and a browser's user-agent, not a browser; records the answers DECLARED robots.txt allow / disallow per crawler declared, not enforced OBSERVED the door DNS · a wall, seen when it names itself in a header · an origin — one status comes back; which of them answered is not recorded reads the declaration knocks declared against, served WHAT THE WIRE RECORDS the status is observed; which hop produced it is not 200 served 402 priced a price named in a payment header, or none · settled off the wire 403 refused a challenge a header from a wall that names itself 429 slowed no answer a registry sellers list a machine-payable price per endpoint reads the listing read by the census observable at the door not observable at the door

Every knock this figure records is the census’s own; the second carries GPTBot’s published user-agent string — the string, not its addresses. From one residential connection, dual-stack — IPv4 geolocates to Luxembourg, IPv6 to Belgium — with the address family per request not recorded. Nothing on this page shows what any other crawler — GPTBot itself included — was served, or whether any named crawler obeyed a directive; robots.txt is not an access control, and a knock served against a declared block is protocol-normal.

30,712 robots.txt read 6,174 doors reached 2,082 · one CDN named itself in a header — presence, not policy listing of 2026-09-23 joined against 49,156 registrable domains of the 50,000-row frame 19 list a price third knock since the 2026-09-06 edition

Fig. 2 context the rules, the door’s stages, and the money
Context the instrument cannot see — except where it says so roles, not organisations · no figures RULES a regulator a copyright opt-out a TDM reservation — read by the census, not scored a content-signal in robots.txt — read, not scored a publisher's own terms a licence page, a tariff · an RSL licence file, linked from robots.txt — read by the census, not scored robots.txt in the census's figure the rule carried at the door as terms THE DOOR'S STAGES DNS resolves the wall a CDN · bot management — the wall named in the census's door the origin what the server answers passes MONEY a payer a lab · an agent’s account the same lab as the crawler in the census's figure a facilitator a payment protocol, or the same CDN as the wall; takes a fee settles the price a 402 names, in the census's figure a publisher owns the origin · and the terms above lists a price in the registry, in the census's figure pays settles a subscription or ads, off the wire a licence, off the wire not observable at the door Context the instrument cannot see — except where it says so roles, not organisations · no figures RULES a regulator a copyright opt-out a TDM reservation — read by the census, not scored a content-signal in robots.txt — read, not scored a publisher's own terms a licence page, a tariff · an RSL licence file, linked from robots.txt — read by the census, not scored robots.txt in the census's figure the rule carried at the door · as terms THE DOOR'S STAGES DNS · resolves the wall a CDN · bot management — the wall named in the census's door the origin · what the server answers MONEY a payer a lab · an agent’s account the same lab as the crawler in the census's figure a facilitator a payment protocol, or the same CDN as the wall; takes a fee · settles the price a 402 names, in the census's figure a publisher owns the origin · and the terms above lists a price in the registry, in the census's figure pays settles a licence, off the wire a subscription or ads, off the wire
How to read these

Every rung is one tile. It states what the census read on this edition, what the registered rule needs and by when, and how close the two are. The arithmetic behind the gap, the registered condition and the field it comes from are behind How this one is measured on each tile.

The mark. The upright is the target. The dot is what the census read on this edition, placed by how many times over or under the target it is, so a dot near the upright is near its target and a dot at the end is nowhere near. The dot turns green when the reading meets the rule; until then an arrow points the way it has to move, where there is room to draw one. A chevron at either end means the reading is off this scale. A rung that reads a rank is drawn on the same geometry and its distance is said in places: a rank is an order, not an amount, so no multiple describes it. The same mark, stacked, is the profile strip at the top of each card — one row per scorable rung, a rung whose rule has no single target drawing none and saying so; where a multiple rests on a count under twenty the count is named instead, because that is the floor this page holds before it states a multiple, and where the target is another live reading rather than a line, the row says so.
Cleared is not reached. Most rules have to hold on several consecutive editions before the rung counts as reached, so a dot past the upright is one edition, not a verdict. Each tile says how many it has: 0 of 3 clearing editions.

One tile, one rung:

  • AT REGISTRATIONthe reading on the day we froze the register. Not a test, and it cannot be missed.
  • OPENregistered, measurable, and not met on this edition.
  • REACHEDmet, on the rule exactly as it was registered. Not in use on this edition.
  • WITNESSa person has to see this happen. No instrument scores it.
  • UNSCORABLEno field the census publishes reads this yet.

Not in use on this edition: missed.

One card, one destination:

  • STANDINGregistered, and nothing has yet met a rule that would retire it.
  • WITHHELDregistered and named, with nothing scored: the census has no field for its rungs yet.

Not in use on this edition: collapsed, arrived and retired.

facilitator — a payment protocol, or the same CDN as the wall; takes a fee.

The register and its readings (/futures.json) are CC BY 4.0 — cite with attribution.

The four destinations

Destination D

The Toll Web

STANDING

The claim registered 2026-09-05, unchanged since

By the first edition on or after 1 Sep 2030, 402 Payment Required will outnumber 403 Forbidden at the doors of the ranked web, and a majority of 402s will carry a protocol-confirmed price.

The Toll Web measures posted prices and never measures a transaction. Whether money moves is witnessed by a person reading a public disclosure, never instrumented. A web where every door quotes a price and nobody ever pays satisfies this destination completely.

Is this happening?

None of the 5 scorable rungs clears its rule on this reading. Comparing each reading with its own target, the closest is D2, 4 times under it, and the first checkpoint of any rung falls in 56 days (D2, 2026-11-18). D6 is left out of that comparison: it reads a rank, and a rank has no multiple. Its kill condition already holds on this reading: if it holds on the first edition on or after 2027-08-18, this destination is dead.

  1. D24× under, on a count of 42026-11-18The 402 starts naming a price
  2. D320× under, on a count of 42027-08-18A quarter of the tills are open
  3. D45.4× under a line that moves2028-09-01402 ≥ 403
  4. D578× under, on a count of 22029-09-01The price is in the protocol
  5. D61,227 places away2029-09-01A top-100 domain quotes a machine price

The day it arrives

Manchester, Friday 28 June 2030

Priya's phone bill has a new page. Not the carrier's — her agent's. Forty-one pence this month: a few hundred reads at a fraction of a penny, three recipes at a penny each, one long-form investigation at four pence, and a line she does not understand for "verification". She has never visited any of the sites. Her agent has, at the door, where each of them quoted a price in a header, and paid.

Fig. D scene who arrives at the door

What the door can tell apart under this destination, and what the census can see of it — roles, not organisations.

a rate card, carried at the door: a floor and a ceiling per read THE DOOR a reader's browser through an agent a verified agent an unverified agent a lab's trainer 402 priced 403 refused priced refused a reader's agent a facilitator a publisher pays fractions, through the agent off the wire — not observed by the census bent by the scene not observable sceneThe Toll Web a rate card, carried at the door: a floor and a ceiling per read a reader's browser through an agent a verified agent a lab's trainer an unverified agent 402 priced 403 refused priced refused the door pays fractions, through the agent a reader's agent a facilitator a publisher off the wire — not observed by the census bent by the scene not observable scene The Toll Web

What would kill it one reading that ends it

registered

at 18 Aug 2027 fewer than 5% of 402s carry a payment header — a year after the switch shipped, 402 is still a politer 403

One condition, on one reading.

  • Twenty times 402s carrying a payment header (4) = 80 is below 402 answers in all (312)holds

The window has not opened: it is the first edition published on or after 2027-08-18.

The readings 8 rungs 5 scorable 0 reached

Asked 6,617 of the 50,000-domain frame (the top 2,000 by rank and every domain whose robots.txt blocks one of 18 tracked crawlers — a set built to over-represent blockers); reached 6,174 — our identified crawler got an HTTP status — which then took the second and third knocks, once each, redirects followed, whether or not the door disallows the crawler presented, and never repeats it. A second instrument, the 422-door identity matrix, presents the same string once more at doors it shares with the census — see /methodology.

Every registered rung is scorable on this edition next checkpoint 2026-11-18 last read 2026-09-23 D1, D2, D3, D4, D5 and D6 read the wire on one crawler identity: GPTBot

Not independent. D2 and D3 read one number — one piece of evidence, not two.

  1. D1 402 is in use
    AT REGISTRATION
    321 402 answers of 7,491 reached domains (4.3%)
    the reading when we registered this ladder
  2. D2 The 402 starts naming a price
    OPEN
    4 402s with a payment header of 312 (402 responses on the wire) (1.3%)
    needs 16 of 312 (5.0%) or more
    checkpoint 2026-11-18 0 of 3 clearing editions ±0 across 8 readings, needs +12
    How this one is measured
    ≥ 5% of 402s carry a payment header, three consecutive editions
    4.13.9registered08-232026-08-26: 408-2608-302026-09-02: 409-022026-09-06: 409-062026-09-09: 409-092026-09-13: 409-132026-09-16: 409-162026-09-20: 409-202026-09-23: 409-23

    8 readings in 10 published editions; the archive carries no value for the other 2; the target, 16, is off this scale. The plot is scaled to the readings, not to zero. Hollow dots were read before the register was frozen and scored nothing. 4 4 4 4 4 4 4 4

    Condition not met this edition — 0 of 3 consecutive editions.
    The reading is 12 short of the target, a gap of 3.8 points on this base.
    One more moves the reading 0.32 points against that gap, so 12 of them would meet it.
  3. D3 A quarter of the tills are open
    OPEN
    4 402s with a payment header of 312 (402 responses on the wire) (1.3%)
    needs 78 of 312 (25.0%) or more
    checkpoint 2027-08-18 0 of 3 clearing editions ±0 across 8 readings, needs +74
    How this one is measured
    ≥ 25% of 402s carry a payment header
    4.13.9registered08-232026-08-26: 408-2608-302026-09-02: 409-022026-09-06: 409-062026-09-09: 409-092026-09-13: 409-132026-09-16: 409-162026-09-20: 409-202026-09-23: 409-23

    8 readings in 10 published editions; the archive carries no value for the other 2; the target, 78, is off this scale. The plot is scaled to the readings, not to zero. Hollow dots were read before the register was frozen and scored nothing. 4 4 4 4 4 4 4 4

    Condition not met this edition — 0 of 3 consecutive editions.
    The target is 19.5 times this reading.
    One more moves the reading 0.32 points against that gap, so 74 of them would meet it.
  4. D4 402 ≥ 403
    OPEN
    402: 312 403: 1,672 of 6,174 reached domains (one base for both counts)
    needs 1,672 or more, which is another reading, not a fixed line
    checkpoint 2028-09-01 0 of 3 clearing editions −6 across 8 readings, needs +1,360
    How this one is measured
    402 answers at least equal 403 answers on the wire
    327.4303.6registered08-232026-08-26: 31808-2608-302026-09-02: 32109-022026-09-06: 31909-062026-09-09: 31809-092026-09-13: 32109-132026-09-16: 31009-162026-09-20: 31109-202026-09-23: 31209-23

    8 readings in 10 published editions; the archive carries no value for the other 2; the target, 1,672, is off this scale. The plot is scaled to the readings, not to zero. Hollow dots were read before the register was frozen and scored nothing. 318 321 319 318 321 310 311 312

    Condition not met this edition — 0 of 3 consecutive editions.
    The target is 5.4 times this reading.
    that figure is not a goalpost: it is dashboard.wire_join.status_mix.403's reading on this edition, and this rung is also met if it falls
    At the fastest movement recorded across 8 published editions (2 of them before the register was frozen, so they are readings and not scores), the gap needs 454 more editions, which at this instrument’s median 4-day gap between editions would be about 2031-09-13. A floor on effort, not a forecast.
  5. D5 The price is in the protocol
    OPEN
    2 402s with a protocol-confirmed price of 312 (402 responses on the wire) (a count, not a rate)
    needs 156 or more
    checkpoint 2029-09-01 0 of 3 clearing editions ±0 across 8 readings, needs +154
    How this one is measured
    ≥ 50% of 402s carry a protocol-confirmed price
    2registered08-232026-08-26: 208-2608-302026-09-02: 209-022026-09-06: 209-062026-09-09: 209-092026-09-13: 209-132026-09-16: 209-162026-09-20: 209-202026-09-23: 209-23

    8 readings in 10 published editions; the archive carries no value for the other 2; the target, 156, is off this scale. The plot is scaled to the readings, not to zero. Hollow dots were read before the register was frozen and scored nothing. 2 2 2 2 2 2 2 2

    Condition not met this edition — 0 of 3 consecutive editions.
    The target is 78.0 times this reading.
  6. D6 A top-100 domain quotes a machine price
    OPEN
    1,327 rank of the highest-ranked confirmed price
    needs 100 or fewer
    checkpoint 2029-09-01 not clearing on this edition
    How this one is measured
    a domain ranked ≤ 100 has a protocol-confirmed price on the wire
    Condition not met this edition — 0 of 1 consecutive editions.
    The reading is 1,227 places away from the target, and the rank has to come down to it. A rank is an order, not an amount, so there is no multiple to state.

Watched, not scored

2 of this destination’s rungs have no instrument behind them: a person has to see them happen, or the census has no field for them yet. They are registered, dated and scored by nobody.

  • D7 Money moves
    WITNESS 2030-06-30
    a facilitator or CDN discloses ≥ 1 billion settled per-crawl payments in twelve months
  • D8 The rate card
    WITNESS 2031-03-31
    a national or EU regulator publishes or endorses a per-crawl price schedule or collective tariff for machine access

Dates, fields and hashes: /futures.json.

Destination A

The End of the Internet of Humans

STANDING

The claim registered 2026-09-05, unchanged since

By the first edition on or after 1 Sep 2033, the ranked web will treat the visitor that trains a model and the visitor that stands in for a reader as different things at the door, and a top-100 domain will answer a plain browser with a machine price rather than a page.

Is this happening?

None of the 3 scorable rungs clears its rule on this reading. Comparing each reading with its own target, the closest is A4, 1.3 times under it, and the first checkpoint of any rung falls in 329 days (A2, 2027-08-18). A2 has no single target to quote.

  1. A2no single target to quote2027-08-18The reader's agent is spared
  2. A32.9× under2028-09-01Discrimination goes mainstream
  3. A41.3× under2028-09-01The door knows who you are

The day it arrives

Lisbon, Tuesday 14 October 2033

Inês is buying a flat. She says so, once, to the thing in her ear on the tram, and by the time she reaches the office it has read a couple of hundred pages from a few dozen sites, paid eleven of them, argued with a bank's agent about a fee, and has three offers for her to look at on a single card. She never sees a website. Nobody she knows under forty has typed a URL this year.

Fig. A scene who arrives at the door

What the door can tell apart under this destination, and what the census can see of it — roles, not organisations.

THE DOOR a reader's browser a reader's agent a lab's trainer a price list, not a page 200 served, priced 403 refused served served refused a reader's agent a facilitator a publisher pays per read; the reader never sees the page — consumer law reaches the purchase, off the wire off the wire — not observed by the census bent by the scene not observable sceneThe End of the Internet of Humans a reader's browser a reader's agent a lab's trainer a price list, not a page 200 served, priced 403 refused served served refused the door pays per read; the reader never sees the page — consumer law reaches the purchase, off the wire a reader's agent a facilitator a publisher off the wire — not observed by the census bent by the scene not observable scene The End of the Internet of Humans

What would kill it three consecutive readings

registered

at 18 Aug 2027 two of the three user-initiated fetchers have risen by as many points as GPTBot (GPTBot having risen at all), or the selective share is at or below its 2 Sep 2026 reading — the ranked web has decided a bot is a bot

Either of two conditions ends it. The rule has to hold on three consecutive editions, counted from 2027-08-18.

  • Both of these must hold
    • At least two of three must hold three hold on this reading
      • ChatGPT-User’s block rate’s move since 2026-09-02 (+0.18 pp) is at or above GPTBot’s block rate’s move since 2026-09-02 (-4.34 pp)holds
      • Claude-Web’s block rate’s move since 2026-09-02 (+0.43 pp) is at or above GPTBot’s block rate’s move since 2026-09-02 (-4.34 pp)holds
      • Perplexity-User’s block rate’s move since 2026-09-02 (+0.15 pp) is at or above GPTBot’s block rate’s move since 2026-09-02 (-4.34 pp)holds
    • GPTBot’s block rate’s move since 2026-09-02 (-4.34 pp) is above zerodoes not hold
  • The selective share (2.75) is at or below its 2026-09-02 reading (2.61)does not hold
crawler2026-09-23n 30,7122026-09-02n 31,488change
ChatGPT-User6.56%6.38%+0.18 pp
GPTBot11.57%15.91%-4.34 pp
Claude-Web6.03%5.60%+0.43 pp
Perplexity-User2.54%2.39%+0.15 pp

The window has not opened: it is the first three editions published on or after 2027-08-18.

The readings 8 rungs 3 scorable 0 reached

Asked 6,617 of the 50,000-domain frame (the top 2,000 by rank and every domain whose robots.txt blocks one of 18 tracked crawlers — a set built to over-represent blockers); reached 6,174 — our identified crawler got an HTTP status — which then took the second and third knocks, once each, redirects followed, whether or not the door disallows the crawler presented, and never repeats it. A second instrument, the 422-door identity matrix, presents the same string once more at doors it shares with the census — see /methodology. Also read on 30,712 (61.4%) domains with a readable robots.txt, of 50,000 domains in the ranked frame.

1 registered rung is not scorable on this edition, and arrival needs all 4 next checkpoint 2027-08-18 last read 2026-09-23 A4 and A7 read the wire on one crawler identity: GPTBot

  1. A1 The line exists
    AT REGISTRATION
    1,474 block training only 108 block search-or-user only of 31,488 domains with a readable robots.txt
    the reading when we registered this ladder
  2. A2 The reader's agent is spared
    OPEN
    ChatGPT-User +0.18pp Claude-Web +0.43pp Perplexity-User +0.15pp GPTBot −4.34pp since 2026-09-02
    several conditions have to hold at once; no single target to quote
    checkpoint 2027-08-18 0 of 3 clearing editions −4.21 across 10 readings
    How this one is measured
    each of ChatGPT-User, Claude-Web and Perplexity-User at most 1.0pp above its 2 Sep 2026 reading while GPTBot is at least 3.0pp above its own
    17.79.8registered2026-08-23: 15.7808-232026-08-26: 15.7508-262026-08-30: 15.8508-302026-09-02: 15.9109-022026-09-06: 15.9309-062026-09-09: 15.9909-092026-09-13: 16.0709-132026-09-16: 11.4209-162026-09-20: 11.5709-202026-09-23: 11.5709-23

    10 readings. The plot is scaled to the readings, not to zero. Hollow dots were read before the register was frozen and scored nothing. 15.78 15.75 15.85 15.91 15.93 15.99 16.07 11.42 11.57 11.57

    Condition not met this edition — 0 of 3 consecutive editions.
  3. A3 Discrimination goes mainstream
    OPEN
    2.75% of 30,712 domains with a readable robots.txt
    needs 8.0% or more
    checkpoint 2028-09-01 0 of 3 clearing editions
    How this one is measured
    ≥ 8% of parsed domains treat at least one crawler differently from another
    Condition not met this edition — 0 of 3 consecutive editions.
    The reading is 5.2 points short of the target.
  4. A4 The door knows who you are
    OPEN
    1,143 of 2,219 refusals on the wire (51.5%)
    needs 1,443 of 2,219 (65.0%) or more
    checkpoint 2028-09-01 0 of 3 clearing editions −507 across 7 readings, needs +300
    How this one is measured
    crawler-specific refusals ≥ 65% of all refusals on the wire
    1860.1931.9registered08-2308-2608-302026-09-02: 165009-022026-09-06: 164909-062026-09-09: 166909-092026-09-13: 166009-132026-09-16: 112309-162026-09-20: 114109-202026-09-23: 114309-23

    7 readings in 10 published editions; the archive carries no value for the other 3; the dashed rule is the target, as this edition solves it. The plot is scaled to the readings, not to zero. Hollow dots were read before the register was frozen and scored nothing. 1,650 1,649 1,669 1,660 1,123 1,141 1,143

    Condition not met this edition — 0 of 3 consecutive editions.
    The reading is 300 short of the target, a gap of 13.5 points on this base.
    One more moves the reading 0.045 points against that gap, so 300 of them would meet it.
    At the fastest movement recorded across 7 published editions (1 of them before the register was frozen, so they are readings and not scores), the gap needs 15 more editions, which at this instrument’s median 4-day gap between editions would be about 2026-11-22. A floor on effort, not a forecast.
    Two halves of one base. This rung and C4 split 2,219 requests the wire refused to the declared crawler between them, so they sum to 100%. Their thresholds do not: reaching this one pushes the other away by the same amount, and they cannot both be reached.

Watched, not scored

4 of this destination’s rungs have no instrument behind them: a person has to see them happen, or the census has no field for them yet. They are registered, dated and scored by nobody.

  • A5 Machines are the majority
    WITNESS 2029-12-31
    automated share of requests ≥ 70% on at least two of three bot-mitigation networks
  • A6 The click dies
    WITNESS 2030-12-31
    AI-referred visits exceed search-referred visits for the top 100 news domains, or Chrome UX Report page views for the top 1,000 fall ≥ 30% from 2026
  • A7 The browser is the stranger
    UNSCORABLE 2031-09-01
    ≥ 10% of reached wire domains answer the browser control with something other than 200
    registered as “the browser control”; the field it reads is the identified-crawler control, which is not published; which control the rung means is unresolved — unscorable under either reading until re-registered amended 2026-09-10
    waits for dashboard.wire_join.control_status_mix
  • A8 The front page is a price list
    WITNESS 2033-09-01
    a top-100 domain answers the browser control with 402 and carries a protocol-confirmed price (feed half, waits for index.price_headline.confirmed_top_rank); Common Crawl records fewer than half of top-1,000 homepages with a rendered HTML body (witness half)
    registered as “the browser control”; which control the rung means is unresolved (see A7); its feed half waits for index.price_headline.confirmed_top_rank, its witness half is not scored by us; unscorable until re-registered amended 2026-09-10

Dates, fields and hashes: /futures.json.

Destination C

The Great Lockout

STANDING

The claim registered 2026-09-05, unchanged since

By the first edition on or after 1 Sep 2031, more than half of the 50,000-domain frame will refuse or wall an identified, signed measurement crawler — the census will be unable to read a majority of the ranked web — while at least 40,000 of those hosts are still alive.

Is this happening?

None of the 5 scorable rungs clears its rule on this reading. Comparing each reading with its own target, the closest is C5, 1.2 times over it, and the first checkpoint of any rung falls in 329 days (C2, 2027-08-18). C2 has no single target to quote.

  1. C2no single target to quote2027-08-18The wall grows
  2. C32.2× under2028-09-01One in four
  3. C41.2× under2028-09-01The door refuses everyone
  4. C51.2× over2029-09-01Alive but unreadable
  5. C86.1× under2031-09-01The census cannot read half the frame

The day it arrives

Aarhus, Sunday 7 September 2031

At 03:01 UTC the census runs, as it has every Sunday and Wednesday since August 2026. By 03:40 the status page updates itself and, for the first time, prints the sentence its author wrote into the builder five years earlier in case it was ever needed: This edition could not read a majority of the frame.

Fig. C scene who arrives at the door

What the door can tell apart under this destination, and what the census can see of it — roles, not organisations.

the wall admits signed or paid identities only; the rules name a verified auditor THE DOOR a reader's agent with an account a lab's trainer an archive crawler a verified auditor 200 served 403 refused served refused a lab a licence, off the wire a publisher the crawl refused; a licence, off the wire off the wire — not observed by the census bent by the scene not observable sceneThe Great Lockout the wall admits signed or paid identities only; the rules name a verified auditor a reader's agent with an account a verified auditor a lab's trainer an archive crawler 200 served 403 refused served refused the door the crawl refused; a licence, off the wire a lab a licence, off the wire a publisher off the wire — not observed by the census bent by the scene not observable scene The Great Lockout

What would kill it one reading that ends it

registered

at 18 Aug 2027 at most 3,000 frame domains refuse the identified crawler and bot-walled hosts are at most 10% of live hosts, with the method unchanged — the walls are not growing

Both conditions must hold together.

  • Frame domains refusing the identified crawler (4,092) is at or below three thousanddoes not hold
  • Ten times bot-walled hosts (5,094) = 50,940 is at or below live hosts (44,127)does not hold

The method clause is registered and not scored: a collapse would be declared on the two measured clauses alone.

The window has not opened: it is the first edition published on or after 2027-08-18.

The readings 8 rungs 5 scorable 0 reached

Asked 6,617 of the 50,000-domain frame (the top 2,000 by rank and every domain whose robots.txt blocks one of 18 tracked crawlers — a set built to over-represent blockers); reached 6,174 — our identified crawler got an HTTP status — which then took the second and third knocks, once each, redirects followed, whether or not the door disallows the crawler presented, and never repeats it. A second instrument, the 422-door identity matrix, presents the same string once more at doors it shares with the census — see /methodology. Also read on 44,127 (88.3%) live hosts, of 50,000 domains in the ranked frame.

Every registered rung is scorable on this edition next checkpoint 2027-08-18 last read 2026-09-23 C4 read the wire on one crawler identity: GPTBot

Not independent. C2 and C8; C2 and C3; C5 and C8 share readings — fewer pieces of evidence than rungs.

  1. C1 The wall exists
    AT REGISTRATION
    3,760 domains refusing an identified crawler asking for robots.txt (403 or 429) of 50,000 domains in the ranked frame (7.5%)
    the reading when we registered this ladder
  2. C2 The wall grows
    OPEN
    4,092 refusing domains of 50,000 domains in the ranked frame (8.2%) or 5,094 bot-walled hosts of 44,127 live hosts (11.5%)
    two separate routes to this one; no single target to quote
    checkpoint 2027-08-18 0 of 3 clearing editions
    How this one is measured
    refused-crawler domains ≥ 10% of the 50,000-domain frame (5,000), OR bot-walled hosts ≥ 15% of live hosts, with the method unchanged
    Condition not met this edition — 0 of 3 consecutive editions.
  3. C3 One in four
    OPEN
    5,094 bot-walled hosts of 44,127 live hosts (11.5%)
    needs 11,032 of 44,127 (25.0%) or more
    checkpoint 2028-09-01 0 of 3 clearing editions +237 across 10 readings, needs +5,938
    How this one is measured
    bot-walled hosts ≥ 25% of live hosts
    5195.94755.1registered2026-08-23: 485708-232026-08-26: 491108-262026-08-30: 501408-302026-09-02: 499309-022026-09-06: 491509-062026-09-09: 497009-092026-09-13: 493409-132026-09-16: 498809-162026-09-20: 503809-202026-09-23: 509409-23

    10 readings; the target, 11,032, is off this scale. The plot is scaled to the readings, not to zero. Hollow dots were read before the register was frozen and scored nothing. 4,857 4,911 5,014 4,993 4,915 4,970 4,934 4,988 5,038 5,094

    Condition not met this edition — 0 of 3 consecutive editions.
    The reading is 5,938 short of the target, a gap of 13.5 points on this base.
    At the fastest movement recorded across 10 published editions (4 of them before the register was frozen, so they are readings and not scores), the gap needs 58 more editions, which at this instrument’s median 3-day gap between editions would be about 2027-03-16. A floor on effort, not a forecast.
  4. C4 The door refuses everyone
    OPEN
    1,076 of 2,219 refusals on the wire (48.5%)
    needs 1,332 of 2,219 (60.0%) or more
    checkpoint 2028-09-01 0 of 3 clearing editions
    How this one is measured
    refusals that also refuse the honest control ≥ 60% of all refusals on the wire
    Condition not met this edition — 0 of 3 consecutive editions.
    The reading is 256 short of the target, a gap of 11.5 points on this base.
    One more moves the reading 0.045 points against that gap, so 256 of them would meet it.
    Two halves of one base. This rung and A4 split 2,219 requests the wire refused to the declared crawler between them, so they sum to 100%. Their thresholds do not: reaching this one pushes the other away by the same amount, and they cannot both be reached.
  5. C5 Alive but unreadable
    OPEN
    30,712 44,127 parsed alive of 50,000 domains in the ranked frame
    needs 25,000 or fewer
    checkpoint 2029-09-01 0 of 3 clearing editions −757 across 10 readings, needs −5,712
    How this one is measured
    readable robots.txt ≤ 50% of the frame while alive hosts ≥ 85% of it
    32191.230070.8registered2026-08-23: 3146908-232026-08-26: 3156008-262026-08-30: 3150008-302026-09-02: 3148809-022026-09-06: 3144109-062026-09-09: 3146909-092026-09-13: 3136409-132026-09-16: 3070209-162026-09-20: 3074209-202026-09-23: 3071209-23

    10 readings; the target, 25,000, is off this scale. The plot is scaled to the readings, not to zero. Hollow dots were read before the register was frozen and scored nothing. 31,469 31,560 31,500 31,488 31,441 31,469 31,364 30,702 30,742 30,712

    Condition not met this edition — 0 of 3 consecutive editions.
    The reading is 5,712 above the target, and it has to fall.
    At the fastest movement recorded across 10 published editions (4 of them before the register was frozen, so they are readings and not scores), the gap needs 9 more editions, which at this instrument’s median 3-day gap between editions would be about 2026-10-20. A floor on effort, not a forecast.
  6. C8 The census cannot read half the frame
    OPEN
    4,092 44,127 refusing alive of 50,000 domains in the ranked frame
    needs 25,000 or more, on one of two routes
    checkpoint 2031-09-01 0 of 3 clearing editions
    How this one is measured
    domains refusing an identified crawler ≥ 25,000 of 50,000, or bot-walled hosts ≥ half of alive hosts, while alive hosts ≥ 40,000 — the unresolved and no-file buckets do not count
    Condition not met this edition — 0 of 3 consecutive editions.
    The target is 6.1 times this reading.
    one way to reach this rung; the registered rule above has another

Watched, not scored

2 of this destination’s rungs have no instrument behind them: a person has to see them happen, or the census has no field for them yet. They are registered, dated and scored by nobody.

  • C6 The archives shrink
    WITNESS 2029-12-31
    Common Crawl's monthly fetched-page count or the Internet Archive's yearly captures for the top 10,000 fall ≥ 30% from 2026, or Common Crawl reports ≥ 25% of fetches refused
  • C7 The default flips for everyone
    WITNESS 2030-12-31
    a CDN applies block-by-default to existing zones, not only newly onboarding ones, and its own radar shows verified-bot requests blocked ≥ 50%

Dates, fields and hashes: /futures.json.

The Machine-Only Web is registered as destination B and withheld: registered and withheld until the registry capture carries index.bazaar_headline.distinct_domains in the feed, a count the census does not publish. No claim, no rungs and no scene are written for it yet, and none is scored.

One dated event

Cloudflare's 15 September default

What changes registered 2026-09-05 takes effect 2026-09-15

From 15 September 2026 Cloudflare blocks bots it classifies as Training or Agent on ad-displaying pages for newly onboarding domains, while Search remains allowed.

No threshold is registered on this card. The Crawl Price Index has no commercial relationship with Cloudflare, TollBit, or any facilitator or vendor named on this page.

The readings either side of it

Cloudflare's change applies only to domains newly onboarding to Cloudflare, so it is not expected to move this frame materially. This split exists to measure whether an infrastructure default is visible in declared policy at all.

edition402s carrying a payment headerdomains blocking machine consumption onlyof domains with a readable robots.txtcontent-signal lines carrying the inherited default
2026-09-064 of 3196.53% of 31,4411,680
2026-09-094 of 3186.56% of 31,4691,692
2026-09-134 of 3216.58% of 31,3641,703
2026-09-164 of 3106.96% of 30,70257
2026-09-204 of 3116.95% of 30,742110
2026-09-234 of 3126.94% of 30,712122

402s carrying a payment header. no mechanism is registered: this column is watched, not predicted

content-signal lines carrying the inherited default. matched on the line as served; a hosting template change would move this count without any publisher deciding anything

These are scenarios, not forecasts. Four ways the crawl economy could go, each written down in advance with the readings that would prove or disprove it. Every number here comes from our published feed with its denominator; nothing written on this page changes a measurement. Some rungs read the same field, so they move together. Registered 2026-09-05; scored by a script against edition 2026-09-23.

Each card carries a short scene of the day it arrives and a diagram of the machinery behind it. Those are fiction, and no number in them is measured.

The register is /futures.json, at commit 89828c7. Registered 2026-09-05, frozen since.

The goalposts sha256 is abde3c2309d2d8da11d82b691e377694025ae1de1ecf77ec39cc8946034b85f5. Per destination it covers the dated claim, the arrival date, the status at registration, the baseline edition, and the collapse rule whole — its date, its expression, its consecutive and its text. Per rung it covers bases, clauses, consecutive, date, expression, field, floor, id, requires and witness. It does NOT cover, on a rung, amendments, condition, count_only, denominator, fields, grouping, method_bound, name, note, reason, second_field, series, shared, unit and witnesses; nor, on a destination, name, order, quote, scene, status, text_sha256 and withheld_reason. Those are words ABOUT a goalpost, or fields the register carries for other reasons, and they can be corrected — dated, at /corrections — without moving one. amendments is on that list, and it is the one that could not be left there: an amendment carrying hold takes a rung out of scoring altogether, so what is scored can change while the hash above does not move. So there is a second hash, and it is not this one. The amendments sha256 is 6bf147001ded2ae157d70172b76d19375b6386d406a0206a4dd2bff4cb803831, over every amendment the register carries — two today, on A7 and A8 — each with its date, its hold and its words, sorted so two machines reading this register compute the same digest. Writing, dating or lifting an amendment moves it. Nothing else does, and it never moves the goalposts hash, which is the whole point: the registered thing stays frozen and undated, and the thing written after registration is checkable on its own. The recipe, as coded: sha(bets.map(text_sha256).concat(ladders.map(text_sha256))), bets first — none is registered on this page — over canonical(goalpostsOfLadder(ladder)); and sha(canonical(amendmentsOfRegister(register))) for the second.