Here is a failure mode almost nobody monitors. A site loads perfectly in every browser, scores green in Lighthouse, passes every uptime check, and returns 403 to the crawlers that feed ChatGPT, Perplexity and Claude. The owner never finds out, because the failure produces no error a human ever sees. The site simply stops existing for the fastest-growing way people ask questions.
The problem is common. Across several hundred domain probes run by Honeyb this year, security layers denying AI user agents at the network edge kept appearing on sites that served browsers normally, and the causes almost always lived in the hosting stack: CDN bot-management rules that ship with AI-crawler blocking switched on by default, WAF policies fingerprinting non-browser TLS handshakes, and rate limits that time out crawlers on JS-heavy pages.
In one memorable case, a fintech was preparing a launch while its edge configuration returned 403 to every major AI crawler. Nobody had decided that; a default had.
This guide walks through the whole check, from knowing which crawlers matter, to testing reachability in five minutes with curl, to fixing each failure at the layer it lives in. No tooling required beyond a terminal, though a free automated check exists for the impatient.
Step 1: know which crawlers you are ruling on
Bot management works best when you know which visitors you are ruling on. These are the documented AI crawlers that matter for visibility, and what each one feeds:
| User agent | Operator | What it feeds |
|---|---|---|
| GPTBot | OpenAI | Model training corpus |
| OAI-SearchBot | OpenAI | ChatGPT search retrieval, the one that gets you cited |
| ClaudeBot | Anthropic | Claude’s crawling and retrieval |
| PerplexityBot | Perplexity | Perplexity’s live answers and citations |
| Google-Extended | Gemini training control (search indexing is separate) | |
| CCBot | Common Crawl | The open corpus many models train on |
The nuance that saves real grief: training and retrieval are different decisions. Blocking GPTBot keeps your content out of future training runs, but blocking OAI-SearchBot removes you from ChatGPT’s live, cited answers, and plenty of sites block both while intending only the first. Decide each deliberately rather than inheriting a vendor default.
Step 2: test reachability from a terminal
Permission is not reachability. A robots.txt that allows a bot says nothing about whether your CDN lets the same bot connect. The test is one line per crawler: fetch your homepage with each documented user agent string and compare the status code against a normal browser fetch.
curl -s -o /dev/null -w “%{http_code}\n” -A “Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)” https://yourdomain.com/
Repeat for each user agent in the table, then once with a normal browser string as the control. A 200 for browsers and a 403 for crawlers is the whole diagnosis.
Watch for the subtler variants too: a connection that drops before any status code points at TLS fingerprinting, and a 200 whose body is an empty application shell points at client-side rendering, which to a crawler on a time budget is the same as nothing. Run the same test against your three most important pages, not just the homepage, because bot rules are often path-scoped.
Step 3: check the permission layer
Now read your actual robots.txt, top to bottom, as each bot would. The common surprises: a blanket Disallow inherited from a staging config, an AI-bot block a security plugin added in an update, or rules that allow the training bots while blocking the retrieval ones, which is the exact inverse of what most businesses want.
Whatever policy you choose, choose it on purpose: block training bots if that is your position, but know that blocking retrieval bots removes you from AI answers entirely.
Step 4: fix each failure at the layer it lives in
| Symptom | Where it lives | Fix |
|---|---|---|
| 403 to AI user agents, 200 to browsers | CDN bot management (often default-on) | Allowlist the documented AI crawlers you want retrieval from |
| Connection drops before any HTTP status | WAF TLS fingerprinting | Exempt verified crawler IP ranges from fingerprint rules |
| Empty content retrieved | Client-side rendering | Server-render the pages that answer questions; hydrate after |
| Crawlers see stale pages | Cache rules keyed to browser agents | Normalise cache behaviour across user agents |
| Everything blocked “for security” | robots.txt plus edge, inherited defaults | Re-decide deliberately, bot by bot |
Most of these are one-setting fixes once located. The locating is the work, which is why the curl pass matters: it tells you which layer to open before you start changing configurations blind.
Step 5: verify the content is liftable once crawlers get in
Reachability gets you retrieved; structure gets you cited. Three quick checks once the connection works.
First, view source and confirm your real content is present in the server-rendered HTML rather than arriving via JavaScript.
Second, confirm a real heading hierarchy, because retrieval chunks content along heading boundaries and a wall of styled divs chunks badly.
Third, consider an llms.txt file at the web root, a plain-text map pointing machines at your canonical pages; some engines read it, none penalise it, and it takes ten minutes.
Step 6: re-check monthly, because defaults change
The uncomfortable truth about this whole checklist is that passing it once proves little. CDN vendors update bot-management defaults between releases, security plugins add AI-blocking toggles in minor versions, and a rule you never set can appear without an announcement. Put the curl pass on a monthly calendar, or automate it in whatever monitoring you already run; it is five minutes that protects a growing share of how customers find you.
For the automated version, Honeyb’s AI visibility checker runs the reachability probe alongside a check of how the engines currently describe your brand, free, and Honeyb re-tests it daily on the full platform, turning each day’s findings into one ranked recommendation.
Why this is worth an afternoon
AI answers are volatile (SparkToro measured the same query changing roughly 70% of the time), which means brands are competing for slots that reshuffle constantly. That competition is winnable only for sites the crawlers can reach.
The failure this guide fixes is silent, common, and entirely under your control as the person who runs the hosting stack. The engines are becoming the first place your next customer hears your name, or does not; your edge configuration decides whether you ever had a chance.
