Cloudflare has announced a major change in how AI bots will access websites. From September 15, AI agent crawlers — the bots that fetch web pages in real time while a person waits for an answer — will be blocked by default on a slice of the web. The company announced the change on July 1.
Cloudflare’s three new AI bot categories
Cloudflare has replaced its single block-AI-bots switch with three separate categories. This gives website owners more control over which AI systems can access their content. According to Cloudflare, the three categories are:
- Search — covers bots that index a page to answer questions about it later
- Agent — covers automated systems acting in real time for a user, including ChatGPT’s fetch bot and browser-driving agents
- Training — covers crawlers that pull content into a model’s weights
The controls went live on July 1 for every Cloudflare customer.
What this means for AI builders
Most of the coverage since the announcement has focused on Google. But the more useful part of this change is what it asks of everyone building AI agents, and what it offers them in return. Website owners can now decide which types of AI traffic they want to allow, rather than having a single on-off switch for all AI bots.
For AI companies and developers building agent-based systems, this means they will need to explicitly request permission to access websites that use Cloudflare. The default setting from September 15 will block agent crawlers, so builders must take action to get access.
How to get permission for AI agent crawlers
Website owners who want to allow AI agent crawlers can do so through Cloudflare’s dashboard. The new controls let them enable or disable each of the three categories independently. This means a site could allow search bots but block training crawlers, or allow agents but block search indexing.
For AI builders, the process involves ensuring their crawlers are properly identified and then working with website owners who use Cloudflare to get permission. Cloudflare has not detailed a specific application process, but the change signals that permission-based access will become the standard.
Our Take: A sensible move for web control
In our view, Cloudflare’s decision to split AI bot controls into three categories is a practical step. The old single switch treated all AI traffic the same, which did not make sense. A search bot indexing content for later answers is very different from an agent pulling pages in real time for a user, and both are different from a training crawler that absorbs content into a model.
This change gives website owners real choice. They can support useful AI services while blocking unwanted scraping. For AI builders, the message is clear: permission matters. The days of crawling any website without asking are ending. Builders who respect these controls and work with site owners will have a smoother path forward.
The September 15 deadline is not far away. Both website owners and AI developers should review Cloudflare’s new settings now to understand what changes they need to make.
