What is AI visibility, and how is it different from SEO?

SEO decides whether your page appears in a list of blue links. AI visibility — also called Answer Engine Optimization or AEO — decides whether an AI assistant like ChatGPT, Gemini, Claude or Perplexity can reach your page, understand it, and name your business in its answer. A site can rank well on Google and still be invisible to AI, most commonly because its robots.txt blocks AI crawlers or its content only renders with JavaScript.

How do I check if ChatGPT can see my website?

ChatGPT reaches websites through crawlers named GPTBot, OAI-SearchBot and ChatGPT-User. If your robots.txt file disallows those user agents, ChatGPT can never read or cite your site. Enter your URL in this tool and the AI Visibility report shows each AI crawler's access individually, along with the exact robots.txt rules blocking them.

Why is my site blocked from AI crawlers when my robots.txt allows them?

A CDN can override your file. Cloudflare's AI Crawl Control feature injects its own managed block into robots.txt, disallowing GPTBot, ClaudeBot, Google-Extended and others before your own rules are read. Check Cloudflare's dashboard under AI Crawl Control and set AI crawlers to Allow. Some hosting firewalls and security plugins block these bots by IP as well.

What is llms.txt and do I need one?

llms.txt is an emerging plain-text file at the root of your site that gives AI models a curated summary of your business, services and key pages, in the same spirit as robots.txt for crawlers. It is not yet an official standard and no engine guarantees it will read one, but it costs nothing to add, very few sites have one, and it lets you state plainly who you are rather than leaving models to infer it.

Is this tool free, and does it need my password or website access?

The scan is free and requires no signup, no password and no access to your site. It reads only what is publicly available: your page HTML, robots.txt, sitemap and llms.txt — the same files any search engine or AI crawler fetches. You provide an email only to unlock the full dashboard.

Does this show my Google ranking position?

No, and any free tool claiming to is guessing. Real ranking positions come from Google Search Console for your own site, or a paid rank-tracking API. What this tool measures is topic match: how clearly a page signals that it is about a given query, scored from where the words appear across your title, H1, headings, copy, URL and schema. A page scoring zero is not in the running at all; seventy or higher is properly set up to compete.

How can I improve my AI visibility score quickly?

In order of impact: unblock AI crawlers in robots.txt and at your CDN; add JSON-LD structured data identifying your business; add an FAQ section answering the exact questions customers ask, marked up with FAQPage schema; make sure your content is present in the raw HTML rather than rendered by JavaScript; and publish an llms.txt file. The tool's report ranks these by the points each one would recover on your specific site.

How many pages of my website does the scan cover?

It scans the page you enter first, then finds every other page it can through your sitemaps and internal links and scores those too, up to 150 pages. Progress is shown live and you can stop at any point and keep the results gathered so far. Each page gets its own AI, SEO and speed score, and the dashboard shows the average across the whole site.

Cloudflare Is Blocking AI Crawlers on Your Site — Here's How to Check and Fix It

By LunieaJuly 22, 2026

If your site is behind Cloudflare, its AI Crawl Control feature can inject a managed block at the top of your robots.txt that disallows GPTBot, ClaudeBot, Google-Extended, CCBot and others — regardless of what your own robots.txt rules say. Your file can read Allow: / while the file crawlers actually download says Disallow: /.

Why this is so easy to miss

Site owners check robots.txt by opening the file in their CMS or SEO plugin, where it looks correct. The injection happens at Cloudflare's edge, so it only appears in what is served to the public internet.

It gets harder still if your own computer has a stale DNS entry pointing straight at your origin server, bypassing Cloudflare. You then see the clean file while every crawler in the world sees the blocked one.

The give-away in the served file is a section marked BEGIN Cloudflare Managed content, listing user agents like Amazonbot, Bytespider, CCBot, ClaudeBot, Google-Extended and GPTBot, each with Disallow: /.

How to check what crawlers really receive

Do not test from your own browser cache. Fetch the file the way a crawler would, from outside your network.

The quickest reliable check is a tool that fetches from its own servers — our free AI visibility checker reports each AI crawler's access individually and quotes the exact rules blocking it. You can also ask someone on a different network to open yoursite.com/robots.txt, or use your phone on mobile data instead of Wi-Fi.

Look specifically for the Cloudflare managed block. If it is there, your own rules further down the file will not rescue you, because a crawler obeys the first matching group for its name.

Turning it off in Cloudflare

Log in to the Cloudflare dashboard and select your domain. Open AI Crawl Control, which sits under the Security or Bots area depending on your plan.

Set AI crawlers to Allow, and switch off the managed robots.txt option so Cloudflare stops injecting its rules into your file.

Changes take effect within minutes. Re-run a scan afterwards to confirm the served robots.txt no longer contains the managed block.

One caveat worth understanding: this feature exists because some site owners genuinely want to keep AI models away from their content. If your business sells content, blocking may be the right call. If your business needs to be recommended, it is not.

Frequently asked questions

Does Cloudflare block AI bots by default?

▾

It depends on your plan and when your zone was created. Cloudflare has progressively made blocking AI crawlers the default for new domains, so many site owners have it switched on without ever having chosen it. Always verify rather than assume.

Will allowing AI crawlers slow my site or raise costs?

▾

In practice the traffic is negligible for most sites — AI crawlers fetch far less aggressively than search engine crawlers. If you serve very large media files, you can allow the crawlers for your HTML pages while disallowing heavy asset directories.

Do other CDNs do this too?

▾

Cloudflare is the most prominent, but similar bot-management features exist at Fastly, Akamai and several WordPress hosts. If your robots.txt looks correct at the origin but crawlers still report a block, check whatever sits in front of your server.