AI crawler is an automated visitor that downloads pages so an AI product can train a model, power a search feature, or retrieve content when a user asks a question. Googlebot indexes for search. GPTBot, OAI-SearchBot, and similar names are different tools with different purposes.

They are not interchangeable. Blocking a training bot does not always remove you from conventional search or from every AI Overview. A blanket block can also hide you from assistants that might cite you (AEO). Robots.txt can ask compliant bots to skip paths and can point them at your sitemap. It is not a lock: a bot can ignore it. Enforcement lives at the CDN, server, or WAF.

A real-life example of an AI crawler

A Midtown law firm’s blog shows up in ChatGPT answers and also in server logs as GPTBot. They may want search-related bots in and training bots out — two rules, not one. A café that lives on “near me” discovery often wants to stay findable, even if a model reads the menu page.

Read robots.txt, check CDN logs, then decide bot by bot. For the sign on the door, see On-Site SEO Guide; for bots that ignore the sign, detect stealth bots.