Free tools · runs in your browser · no signup
AI crawler log analyzer
Check if GPTBot, ClaudeBot, PerplexityBot and other AI bots crawled your site. Paste your access logs to see hits, pages and status codes for 24 known crawlers. Nothing is uploaded.
Your logs never leave this page: everything is parsed in your browser and nothing is uploaded.
Running more than one page? The RankRoot Traffic Boost audits your full site and generates fix files for every page, for $49 once.
How to read the results
Each AI bot has one job, so which bots show up tells you how AI tools use your site. The purpose shown for each bot comes from its operator's documentation:
- AI search index (OAI-SearchBot, Claude-SearchBot, PerplexityBot and others) builds the index AI assistants search and cite. Seeing these is the best sign you can appear in AI answers.
- User-triggered fetcher (ChatGPT-User, Claude-User, Perplexity-User and others) fetched a page because a person asked an AI assistant about it.
- AI training crawler (GPTBot, ClaudeBot, meta-externalagent, CCBot and others) collects content that may be used to train models.
Lots of 403, 429 or 5xx status codes mean the bot reached your server but was refused or failed. 404s show URLs bots asked for that don't exist, often old or broken links. To control which bots may crawl, use the AI robots.txt generator. If an AI bot never shows up at all, a CDN or firewall may be blocking it: test that with the AI crawler checker. Every bot is described in the AI crawler list.
Where to find your access logs
On your own server, nginx usually writes to /var/log/nginx/access.log and Apache to /var/log/apache2/access.log or /var/log/httpd/access_log, depending on the distribution. Hosting platforms and CDNs keep request logs in their dashboards or log exports. To shrink a big file first, keep only bot lines: grep -iE "bot|crawl|-user|meta-external" access.log.
FAQ
Are my logs uploaded anywhere?
No. The analyzer runs entirely in your browser. Pasted lines and opened files are parsed on your device and are never sent to RankRoot or anyone else.
Can a user agent be faked?
Yes. Any client can send a user agent such as GPTBot, and Common Crawl warns that other crawlers sometimes pretend to be CCBot. To confirm a request is genuine, check its source IP against the list the operator publishes (OpenAI, Anthropic, Perplexity, Amazon, DuckDuckGo and Mistral publish IP lists) or use the reverse DNS check that Google, Microsoft, Apple and Common Crawl document.
Why don't I see Google-Extended or Applebot-Extended in my logs?
They are robots.txt control tokens, not crawlers, so they never send requests. Google says crawling is done with its existing user agents, and Google-Extended only controls whether that content may be used for Gemini training and grounding. Applebot-Extended does the same for Apple's model training.
GPTBot visits my site but OAI-SearchBot doesn't. What does that mean?
They are separate OpenAI bots. GPTBot crawls content that may be used to train OpenAI's models. OAI-SearchBot surfaces pages in ChatGPT search, and ChatGPT-User fetches a page when a ChatGPT user asks about it. Check that your robots.txt and firewall allow OAI-SearchBot if you want to be cited in ChatGPT search.
Which log formats work?
Any line that contains the user agent. That includes nginx and Apache combined logs and JSON log lines from Cloudflare, Vercel or Netlify. Paths, status codes, IPs and timestamps are read when the format includes them.
More free AI visibility tools
AI crawler checker
See which AI bots (OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and more) can reach your site, including CDN and firewall blocks.
llms.txt generator
Paste your URL and get a ready-to-publish llms.txt built from your homepage, plus AI-friendly robots.txt and schema.
Schema markup generator (JSON-LD)
Build valid JSON-LD for Organization, SoftwareApplication, Product, FAQPage and WebSite. Fill in a form, copy the script tag. Free, runs in your browser.
AI robots.txt generator
Choose which AI crawlers may reach your site, grouped by search, user-triggered and training bots, with one-click presets. Copy a ready robots.txt.
FAQ schema generator
Paste questions and answers, get an HTML FAQ section plus FAQPage JSON-LD that AI answer engines and search can read. Free, no signup.
llms.txt validator
Check an llms.txt against the llmstxt.org format by URL or by pasting it: H1, summary, H2 link lists and the Optional section, with line-by-line fixes.
RankRoot Traffic Boost
Full-site audits and whole-site fix files: robots.txt, llms.txt and schema for every page. $49 once, no subscription.