Cloudflare reports an OpenAI crawler pilot indexed new pages

On 15 October 2026, Cloudflare published Cloudflare and OpenAI: making AI discovery better for everyone. The byline is Matthew Conroy and Jack Galilee of Cloudflare, with Henry Aspegren and James Balfour of OpenAI. The post says Cloudflare and OpenAI launched a research pilot in July 2026 to test whether signals from Cloudflare’s network could help ChatGPT users discover new content. It calls the result early, and it says the pilot has brought previously unknown pages into OpenAI’s search index.

What Cloudflare reports

The figures below are Cloudflare’s. The post says the pilot used a participating cohort of about 1% of Cloudflare’s total network, and that it enabled OpenAI to discover and index hundreds of millions of new web pages. It says OpenAI then independently evaluates and ranks how those pages appear, based on relevance and helpfulness. It does not publish OpenAI’s ranking method.

The post describes the signals in examples, not as a field list. It says automated traffic can make a missing page look interesting, that some sites redirect almost any address to the homepage, and that some addresses lead to copies of the same content. Filtering, it says, keeps a crawler off pages that do not exist or that redirect elsewhere. Change signals say when a page is worth another visit. Cloudflare is also testing signals that flag a potentially harmful page, such as a fake login form on an otherwise legitimate site, without flagging the whole domain. It says early results suggest those signals can identify pages that should stay out of an index.

On what is not shared, the post says: “This research uses metadata about the web for search and discovery, rather than sharing page content for model training.” It also says a signal about a page does not grant permission to access it. The post does not say that user data, request logs, or identifiers are excluded. It does not name a customer plan for the 1% cohort, and it does not say how a site entered that cohort. “Participating” is the word it uses.

The control it does name is AI Crawl Control. The docs page opened with the post says the product monitors how AI services access a site, can set allow or block rules for individual crawlers, and “works automatically on all Cloudflare plans.” That is a control on crawler access. The 15 October post does not say this switch was how the 1% cohort was chosen.

What OpenAI has said

The 15 October post is co-signed by two OpenAI authors and does not contain a separate quotation from them. The quotation on record is from the 8 July 2026 press release that announced the pilot. Nick Ryder, VP of Research at OpenAI, said: “Up-to-date information is important for delivering accurate answers to people using ChatGPT. Piloting this with Cloudflare will allow us to explore whether network-level insights can help us discover content more efficiently.” That release lists example signals as content freshness, traffic quality, and actual page changes, from “participating websites.” It is the July announcement, not a new measurement in the October results post. Matthew Prince’s line in the same release is that sharing network signals could make AI search more efficient.

Crawl policy beside this pilot

Cloudflare’s other writing on crawlers is stricter about access than this discovery note. On 1 July 2025, Matthew Prince wrote that Cloudflare was changing the default to block AI crawlers unless they pay creators. The Pay Per Use post separates that idea from Pay Per Crawl, which it says charges for access, and from Pay Per Use, which pays for a later use the buyer reports. The earlier account of Pay Per Use is that beta. The 15 October post does not say the discovery pilot changes those products, and it does not state a price, a revenue share, or a contract term for the signals.

The confirmed piece is the one Cloudflare published: a July pilot, about 1% of the network, discovery metadata rather than training copies of pages, OpenAI ranking on its own, and a page count Cloudflare attributes to the pilot. Who was in the cohort, and whether any user data moved, are not in the post.