Robots.txt is a plain-text file at the root of a site (yoursite.com/robots.txt) that gives crawlers instructions: which paths they can fetch, and which they should leave alone.
It does not hide pages from people, and it is not a security lock. It is a courtesy sign for bots. A common use is pointing crawlers to your XML sitemap while blocking admin, cart, or duplicate-filter URLs that waste crawl time. Blocking the wrong folder can keep useful pages out of search.
A real-life example of robots.txt
Think of a shop with a “staff only” door. Customers can still walk in if they know the way; the sign just tells delivery drivers not to wander the stockroom. Search engines treat Disallow: /wp-admin/ the same way: skip the back office, index the storefront.
Pair robots.txt with a clean web architecture and an up-to-date sitemap. AI crawlers (training vs. search vs. retrieval) need their own rules — they are not the same as Googlebot. For on-page hygiene that actually earns clicks, see On-Site SEO Guide.