AI Crawler Log Analysis: Verify What Bots Actually Fetch

Your robots.txt is a statement of policy. Your server logs are a statement of fact. If you want to know whether AI crawlers are actually reading your site - which pages, how often, and which bots - the only ground truth is the access log, and reading it takes about twenty minutes with grep and awk. Everything else, including a permissive robots.txt and a green checkmark in an audit tool, tells you what should happen, not what did.

This post covers the user agents worth searching for, the commands to pull them out of a standard access log, the spoofing caveat that makes raw user-agent counts unreliable, and - the part most guides skip - how to interpret what you find, including the diagnostic meaning of finding nothing at all.

The user agents worth grepping for

Six agents cover the fetch types that matter, and the distinction between them matters more than the list:

• GPTBot - OpenAI's training-and-index crawler. Systematic, broad, revisits over time. Documented with its published IP ranges at platform.openai.com/docs/bots.
• OAI-SearchBot - OpenAI's search crawler, building and refreshing the index behind ChatGPT Search. Its pattern sits between a classic crawl and live fetching.
• ChatGPT-User - fires when a ChatGPT user's session causes a live fetch of your page. This is the closest thing logs offer to "a human, via ChatGPT, read this page just now".
• ClaudeBot - Anthropic's crawler; Anthropic documents its bots and how to control them in its docs and help center.
• Claude-User - Anthropic's on-demand fetcher, the Claude-side equivalent of ChatGPT-User: a user's question triggered a retrieval of your URL.
• PerplexityBot and Perplexity-User - Perplexity's index crawler and its query-time fetcher, both documented with verification details at docs.perplexity.ai/guides/bots.

One absence to note: Google-Extended never appears in logs. It is a robots.txt directive that governs what Googlebot's fetches may be used for, not a user agent that fetches anything - so do not grep for it and conclude Google's AI ignores you. The full user-agent reference covers the longer tail.

Pulling the data with grep and awk

Assume a standard combined-format access log at /var/log/nginx/access.log. First, the overview - hits per AI agent:

grep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User' \
  /var/log/nginx/access.log \
  | grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|PerplexityBot|Perplexity-User' \
  | sort | uniq -c | sort -rn

Then the question that actually pays: which pages is a given bot fetching, and what status codes is it getting?

grep -i 'GPTBot' /var/log/nginx/access.log \
  | awk '{print $9, $7}' | sort | uniq -c | sort -rn | head -40

The $9, $7 pair is status code and URL path in combined format; adjust the field numbers if your log format differs. Two more variants earn their keep. Daily trend, to see whether crawling is ramping up or has stopped:

grep -i 'ClaudeBot' /var/log/nginx/access.log \
  | awk -F'[' '{print substr($2, 1, 11)}' | uniq -c

And an error hunt across all AI agents, because a bot that meets 403s or 429s learns to come back less:

grep -iE 'GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot' /var/log/nginx/access.log \
  | awk '$9 >= 400 {print $9, $7}' | sort | uniq -c | sort -rn

If you are behind a CDN, your origin log is not the whole story - the CDN answers many requests without touching origin, and its bot analytics (Cloudflare and friends expose these) or raw log exports are the complete record. Same analysis, different tap point. And whichever tap you use, watch the status codes with fresh eyes: a WAF serving challenge pages returns 403s to bots that your robots.txt cheerfully allows - policy and fact disagreeing in exactly the way only logs reveal.

The spoofing caveat: user agent is a claim, not an identity

A user-agent string is self-reported. Anyone's scraper can call itself GPTBot in the hope of inheriting your allowlist, so raw UA counts are an upper bound on genuine bot traffic, not a measurement. The vendors know this, which is why the major ones publish machine-readable IP ranges: OpenAI's are linked from its bot documentation, Perplexity publishes ranges in its bot docs, and Anthropic documents its crawler and contact points in its help center.

Verification is one command per suspicious IP. Reverse DNS is the quick sniff test:

host 20.171.207.1

A genuine hit resolves into the vendor's published infrastructure and matches their published ranges; a "GPTBot" request from a residential IP or an unrelated cloud tenant is an impostor. For ongoing rigor, script the published JSON range lists into your log pipeline and tag each AI-claiming request verified or unverified. In practice, spot-checking the top IPs behind each UA gets you most of the confidence for a fraction of the effort. This is also exactly the distinction to remember with checker tools, including ours: a robots.txt checker verifies your declared policy toward each agent, while your logs verify actual fetches. You want both, and they answer different questions.

Training crawls versus user fetches: read the lanes separately

The most useful mental model for AI bot logs is two lanes with different meanings.

Lane 1 - index and training crawls (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot). These are systematic: broad coverage, sitemap-driven discovery, revisits spread over days. Healthy lane-1 traffic means your content is entering the corpora and indexes that future answers draw from. It is a leading indicator - necessary groundwork, no guarantee of citations, in the same way Googlebot crawling never guaranteed rankings. What the crawlers do with a page once fetched is its own topic.

Lane 2 - on-demand user fetches (ChatGPT-User, Claude-User, Perplexity-User). These are event-shaped: a human asked something, the engine decided your specific URL might answer it, and fetched it within seconds of the question. Lane-2 hits are the more exciting signal - each is a real user query that reached your content, even if no click ever lands in your analytics. The pages these agents fetch repeatedly are the pages answer engines already treat as sources; study them, and note that this log signal captures exposure that referral tracking structurally cannot see.

Skewed ratios are diagnostic too. Heavy lane 1 with zero lane 2 means you are being ingested but not yet selected as an answer source - a content and citation-worthiness problem, not a crawl problem. Lane 2 without much lane 1 happens when your robots.txt blocks training crawlers but permits user fetches, a deliberate stance some sites choose; whether to allow AI crawlers at all is a real decision with real trade-offs.

What "no AI bot hits" actually means

Finding nothing is a finding. Work the causes in order:

• Access is blocked. robots.txt disallows, or a WAF/bot-management layer challenges the agents before they reach origin. Check your CDN's bot settings, not just the text file.
• You are looking at the wrong log. CDN-served requests never hit origin; check the edge logs.
• Discovery has not happened. New or poorly linked sites simply have not been found. Sitemaps and crawlable internal linking shorten the wait.
• The bots see no reason to return. Crawlers allocate attention; thin, stale, or duplicate-heavy sites earn sparse revisits.

Only after ruling those out should you conclude the bots genuinely are not interested yet - and each cause has a different fix, which is exactly why the log check comes first.

Frequently asked questions

How far back should I analyze logs?

Thirty days is the useful minimum. Index and training crawlers revisit on multi-day cycles, so a 48-hour window routinely shows nothing on a site that gets crawled perfectly well weekly. If your logs rotate every 7 days, aggregate the rotated files - zgrep works on the compressed archives - or raise retention before drawing conclusions.

Do AI bot hits mean I am being cited?

Not directly. Index and training crawls mean ingestion; only citation checks tell you about selection. The strongest log-side proxy is repeated on-demand fetches - ChatGPT-User or Perplexity-User hitting the same URL from many distinct queries means engines keep choosing that page as answer material. To close the loop you have to query the engines and see what they cite.

Can I see AI bot activity without server access?

Partially. CDN dashboards (Cloudflare's bot and crawler analytics, for example) show AI crawler traffic without touching a shell, and some hosts expose access logs through their control panels. What you cannot do is see bot fetches in GA4 or any JavaScript analytics - crawlers do not execute your analytics tags, so the log layer, wherever you tap it, is the only place this traffic exists.

Should I block unverified requests claiming to be AI bots?

Blocking verified-fake traffic is reasonable - a request claiming to be GPTBot from outside OpenAI's published ranges is lying and deserves nothing. Be careful with automation, though: vendors add ranges, and an out-of-date verification list silently blocks the genuine article. Fail open for unknown IPs on allowed UAs unless you are under active scraping pressure, and refresh published ranges on a schedule.

Policy, then fact, then fixes

The full verification chain has three links: what your robots.txt and WAF declare, what bots actually fetch, and what engines ultimately cite. Citevera's free AI crawler access checker nails down the first link for any URL - honestly labeled as a policy check, because no external tool can watch your logs for you. The grep patterns above give you the second. For the third, a Citevera audit scores whether the pages being fetched are worth citing once they arrive - because the log file can prove the bots came, but not that they found anything worth quoting.