Take Control of AI Crawlers on Cloudflare: A Step-by-Step 2026 Tutorial

Take Control of AI Crawlers on Cloudflare: A Step-by-Step 2026 Tutorial

On September 15, 2026, Cloudflare stopped treating automated AI traffic as a single bucket and split it into three classes: Search, Agent, and Training. The same release shipped defaults for each class that apply to existing zones automatically, which means many site owners now have an AI policy they never consciously chose. This tutorial is the twenty-minute version of getting that control back: audit what is actually hitting your origin, pick a policy per class, publish it in two places, and prove with a real request that the policy is doing what you think.

Step 0: Know which of the three AI classes is hitting you

The three classes are not interchangeable, and collapsing them into one rule is what broke most crawler blocks in 2024 and 2025. They mean different things:

• Search - a crawler fetches a page so a later human question can be answered with a citation back to you. Noisy in logs, valuable in traffic.
• Agent - a delegated request that acts: filling a form, checking stock, comparing prices for a real person who is waiting on the answer right now.
• Training - bulk collection of your pages to build or fine-tune a model. There is no reader on the other end of the request.

Before you change a single setting, pull the last 30 days of bot traffic and group it by AI class in the analytics view. Write down three numbers: percent Search, percent Agent, percent Training. If Training is 55 percent of the total and Search plus Agent is 45 percent, a blanket block throws away the traffic that sends you visitors. Those three numbers drive everything below.

Step 1: Decide your policy per class

You only have four actions per class: allow, allow with a rate limit, challenge, or block. A policy that holds up for most content and commerce sites looks like this. Search: allowed everywhere, no rate limit, because indexing is the point. Agent: allowed on browse and GET, rate limited and challenged on POST, so a scraping loop cannot pretend to be a checkout flow. Training: blocked on anything you sell or license, allowed on pages you are happy to be quoted from.

Rule of thumb: block the collection of the value you monetize, allow the distribution of the value you want spread. A tutorial page is distribution; a paid dataset behind a login is collection.

Write the policy as a three-row table before you touch any control panel, and keep it next to the configuration so the next person inherits the reasoning and not just the settings.

Step 2: Set the zone-level controls

Now translate that table into enforcement. The path in the dashboard is Bots, then the AI crawler panel, and the workflow is the same for each class.

1. Open the zone and go to the AI crawler controls. 2. For the Training row, set the action to block and scope it to the paths you monetize rather than the whole hostname. 3. For the Agent row, set the action to managed challenge and attach a rate limit to POST. 4. For the Search row, leave it on allow and check no inherited rule upstream overrides it. 5. Save, then confirm the change is stamped in the audit log with your account and a timestamp.

If blocking outright makes you nervous, ship the same rules in log-only mode for 48 hours and read what would have been challenged. The same policy can also be written as a custom expression, which helps when one class needs path-level nuance:

(cf.bot_management.verified_bot_category eq 'ai_training')
and not http.request.uri.path starts_with '/docs/'
and not http.request.uri.path starts_with '/marketing/'

Treat that expression as illustrative: field names move between versions, so confirm the exact identifiers in the rule builder. What matters is the shape: one class, one path exception list, one action.

Step 3: Publish your terms in robots.txt

Cloudflare controls decide what gets through the edge. Content Signals in robots.txt declares what you permit once a crawler is through. The two are complements, not substitutes: edge rules bind everyone, robots.txt binds crawlers that play by the rules, and the well-behaved ones do. Adding the signals takes one line to the group you already have:

User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=no, ai-train=no

# scoped variant, for a site that mixes free and licensed content:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /blog/
Allow: /docs/
Disallow: /datasets/
Disallow: /api/

Path scoping is what makes that readable: crawler and auditor alike can tell which pages carry which signal.

Keep the declared sections consistent with what the edge rules enforce. A robots.txt that promises training is blocked while the WAF serves 500 bulk requests an hour is worse than no declaration, because it hides the real state of your site.

Step 4: Verify what you shipped

Both checks read the live site, not the dashboard. First, confirm the declaration is served:

curl -s https://yourdomain.com/robots.txt | grep -i content-signal

The command must echo back the signal line you wrote. Nothing printed means a cached or stale file is being served and the declaration is not live.

Then check that enforcement matches the declaration by requesting a monetized path with a training-class user agent and comparing it with a search-class agent:

curl -s -o /dev/null -w '%{http_code}' -A 'training-crawler-test/1.0' https://yourdomain.com/datasets/
curl -s -o /dev/null -w '%{http_code}' -A 'search-crawler-test/1.0' https://yourdomain.com/blog/

Expect a blocked status such as 403 on the first and a 200 on the second. If both return 200, your path scope is wrong. If both return 403, you have blocked search by accident, the most expensive error in this whole tutorial. Finish by opening the security event log and confirming the two test requests appear there with the class the dashboard assigned them. That last step is what separates a configured policy from a verified one.

Step 5: Monitor and re-check monthly

A policy is a living thing, not a launch task. Put a recurring reminder on the calendar and do four things: re-pull the class split, check the challenge rate for Agent traffic, scan for new crawler categories you have not classified, and re-read your robots.txt against the edge rules. If Agent traffic has grown past a fifth of total automation and your challenge rate is climbing, loosen the challenge on GET before it starts breaking real checkouts. Keep a short allowlist for the agents you depend on, such as your uptime monitor and your payment provider, so a class-level block never takes out infrastructure you rely on.

Common mistakes

• Blocking Search by inheritance. An account-level block silently overrides the allow you set on the zone. Always verify from outside with a test request.
• Treating robots.txt as enforcement. It is a declaration, not a wall. Anything that ignores it walks straight through, which is exactly why the edge rules exist.
• Setting it once and forgetting. Crawler categories get renamed and expanded several times a year. A policy from last spring is a guess today.
• No log-only dry run. Shipping a hard block straight to production is how legitimate agent traffic and your own monitoring get caught in the blast radius.

The Road Ahead

Expect the three-class split to get finer rather than coarser. A reasonable next step for most teams is to treat the taxonomy as a budget: decide how much of your catalog you are willing to give away to each class per month, then enforce that number instead of re-litigating the whole policy every time a new crawler shows up in the logs. Teams that do this treat AI traffic as a distribution channel with a cost.

The practical sequence is short: measure the classes you actually receive, decide an action for each one, enforce it at the edge, declare it in robots.txt, and verify both with a real request from outside your network. Once that loop exists, every future crawler announcement becomes a five-minute classification decision instead of a fire drill.