Cloudflare and OpenAI Publish First Results from Joint Discovery Pilot for the Agentic Web

Cloudflare and OpenAI today published the first results from a joint research pilot launched in July 2026, and the findings confirm that network-level signals can dramatically improve how AI systems discover web content. Using a participating cohort of approximately 1% of Cloudflare’s total network, the pilot enabled OpenAI to discover and index hundreds of millions of previously unknown web pages for ChatGPT Search.

Why discovery matters now

As more people turn to AI for answers — booking tables, applying for passports, troubleshooting appliances — the discoverability of web content takes on new importance. An installation guide, a first-hand product review, or a local restaurant menu can only help someone if an AI system finds it, understands it, and cites it accurately. In the previous version of the web, discoverability meant climbing to the top of a search engine results page. In the agentic web, AI systems need to find the right sources, extract useful information, and connect people with relevant pages, whether through an answer, a citation, or a referral to the website itself.

The pilot addresses a fundamental challenge: the long tail of specific, useful pages that traditional link-based crawling misses. Someone fixing a washing machine needs the manual for their exact model. Someone applying for a passport needs the current requirements and fees. These pages can be immensely valuable even if few other sites link to them.

How the pilot works

Cloudflare’s global network, spanning 335+ cities and powering roughly 20% of the Internet, provides a unique vantage point on web activity. The pilot feeds metadata — traffic patterns, request characteristics, and domain behavior — into OpenAI’s search crawler. OpenAI then independently evaluates and ranks pages based on relevance and helpfulness, linking back to sources so people can explore the original content.

Crucially, this is metadata for search and discovery, not page content for model training. A signal about a page does not grant permission to access it, and domain owners’ choices are respected throughout the pipeline.

Filtering noise before the crawl

Finding a URL is only the first step. The teams learned that automated traffic can easily make a nonexistent page look interesting. Bots repeatedly request files that do not exist. Some sites redirect almost any address to their homepage. Other addresses lead to copies of the same content. Recognizing these patterns lets the system cut wasted work before a crawler ever fetches the page — improving efficiency for both the crawler and the site owner.

The pilot is also testing signals that flag potentially harmful pages without penalizing the whole domain. A legitimate site can have a single compromised page, such as a fake login form planted by an attacker. Flagging just that one page keeps people safe while leaving the rest of the site in the index.

Change signals reduce unnecessary crawling

Keeping a search index current by re-fetching every page more often would be expensive for everyone. For a website owner, a crawler that repeatedly downloads an unchanged page adds load without bringing any benefit. The pilot’s change signals tell crawlers when a page is worth another visit and when it is not, reducing unnecessary crawling while staying current with genuinely updated content.

There is a balance to strike. Finding new pages can initially increase crawling as those pages are evaluated and indexed. The goal is continuous discovery while eliminating the work that wastes resources on both sides.

Cloudflare’s expanding AI infrastructure role

The pilot is part of a broader pattern at Cloudflare, which has been building AI infrastructure at multiple layers. The same blog listing shows the company also announced the Void deployment SDK, published research on malicious bot migration during the Yandex outage, and shipped on-demand CPU and memory profiling for Workers in just the past week. Earlier this month, Cloudflare acquired Deno to bring Workers and Durable Objects to self-hosted environments, launched its Clef-omni multimodal decision model, and introduced Web Search API via AI Gateway. The discovery pilot with OpenAI fits into a strategy where Cloudflare’s network position makes it a natural layer for AI infrastructure — not just serving content, but helping AI systems find and understand it.

What site owners and agent builders should watch

For Automater readers running agent fleets that depend on accurate, current web information, the implications are direct. Better discovery means agents drawing on ChatGPT Search will find more relevant, timely sources. For those managing their own sites, Cloudflare’s AI Crawl Control provides a concrete lever: decide how AI systems access your content, distinguish between search indexing and training use, and manage the load from AI crawlers.

The pilot also signals where web infrastructure is heading. As discovery shifts from human browsing to AI retrieval, the signals that matter change. Link graphs give way to network telemetry. Page rank gives way to relevance and freshness. Site owners who understand this shift can make informed choices about being found in the agentic web — alongside broader developments like Google’s SynthID detector going public, which shows how AI verification and discovery infrastructure are maturing in parallel.

The next phase of the pilot will build on these results: improving page selection, testing faster and more reliable change signals, and measuring what happens after content is discovered — including how better discovery translates into real business outcomes for site owners. Cloudflare and OpenAI say they will continue sharing what they learn.