Free SEO Tool
AI Crawler Check
Can ChatGPT, Claude, and Perplexity actually read your site? This checks your robots.txt against 17 AI crawler policies, your llms.txt discovery layer, and scores AI-readiness 0–100. Free, no sign-up.
# Prefer the terminal? Same audit, open source
pip install zens-ink
zens-ink ai_crawler_audit https://yoursite.com
What the score means
AI-search ready
AI search crawlers can reach you and the llms.txt layer is in place. AI engines can cite and quote your pages.
Partial visibility
Some crawlers are blocked or the llms.txt layer is missing. Usually fixable in minutes with a robots.txt edit.
Invisible to AI search
AI-search crawlers are blocked by robots.txt — ChatGPT, Claude, and Perplexity cannot cite you. Often an accident from disallowing everything.
FAQ
How do I check if AI crawlers can read my website?
Enter your domain above and hit Check. The tool fetches your robots.txt, matches the policies of 17 AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more), checks your llms.txt discovery layer, and scores AI-readiness from 0 to 100. Free, no sign-up.
Which AI crawlers matter for SEO?
AI search crawlers matter most: OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot, and Googlebot (AI Overviews grounding). Blocking these means AI engines cannot cite your pages. Training crawlers like GPTBot or CCBot affect model data, not search visibility.
What is a good robots.txt policy for AI crawlers?
A common balanced setup: allow AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot), allow Googlebot, and decide on training crawlers based on your IP strategy. Add Content-Signals like ai-train=no, search=yes to opt out of training without opting out of search.
What is llms.txt and do I need it?
llms.txt is a markdown file at your site root that lists your key pages for AI engines. It is not a ranking factor, but it helps LLM-based search and assistants discover your content depth. The zens-ink llms_gen tool generates one from your sitemap for free.
Is this AI crawler checker really free?
Yes. The web version runs on this page with no sign-up, and the same audit ships in the open-source zens-ink CLI (`zens-ink ai_crawler_audit yoursite.com`) under MIT license, so you can run it in CI or from your terminal.