What is AI Crawler?
An AI crawler is an automated bot — such as OpenAI's GPTBot or Perplexity's PerplexityBot — that fetches and reads web pages to gather training data or real-time information for an AI system, distinct from traditional search engine crawlers like Googlebot.
AI crawlers identify themselves via a user-agent string, which site owners can allow or block in robots.txt the same way they would any other crawler. A page can have excellent AEO and GEO structure and still never be read, mentioned, or cited if the relevant AI crawlers are blocked from reaching it — crawlability is a prerequisite underneath both disciplines, not an optional extra.
Reviewing raw server access logs for known AI-crawler user agents is one of the few ways to directly confirm whether a specific AI system is actually fetching a given page, since that activity doesn't show up in client-side analytics tools the way a human visit does.