How robots.txt works, with a file builder
robots.txt is a short text file at the root of a site that asks crawlers to stay away from certain paths. Well-behaved crawlers follow it. It is a request, not a lock.
Where it lives and how it is read
The file must be at exactly /robots.txt on each host. A file on the www host does not cover the bare domain or a subdomain; each needs its own. Crawlers read it before fetching other pages and cache it for a while.
The file is made of groups. Each group starts with one or more User-agent lines naming crawlers, followed by rules:
User-agent: *
Disallow: /drafts/
Disallow: /search
Allow: /drafts/press-kit.html
Sitemap: FULL-ADDRESS-OF-SITEMAP
How matching works
- A crawler obeys the group whose
User-agentmatches its own name most specifically, and falls back to*. - Rules match from the start of the path.
Disallow: /searchblocks/search,/search?q=xand also/searchlight.html. Add a trailing slash when you mean a folder. - When an
Allowand aDisallowboth match, major crawlers pick the more specific rule, meaning the longer one. *matches any characters and$marks the end of the URL, soDisallow: /*.pdf$blocks PDF files.- An empty
Disallow:line means nothing is blocked.
robots.txt builder
The builder adds a leading slash to paths that lack one and drops duplicates. It runs in your browser only. Save the output as a plain text file named robots.txt.
What robots.txt cannot do
It does not hide anything. The file is public, so listing /secret-plans/ advertises that folder to anyone curious. Bad bots ignore it completely. And a blocked page can still appear in search results as a bare link if other sites link to it, because the crawler never reads the page to discover it should not be shown.
| Goal | Right tool |
|---|---|
| Save crawl effort on endless filter or search URLs | robots.txt Disallow |
| Keep a page out of search results | noindex meta tag or header, and do not block the page in robots.txt |
| Keep content private | A password or other access control on the server |
Common mistakes
- Shipping
Disallow: /from a staging site to production, which asks every crawler to skip the whole site. - Blocking CSS and JavaScript folders, so search engines cannot render pages properly.
- Combining Disallow and noindex on the same page; the crawler never sees the noindex.
Questions
Is a missing robots.txt a problem?
No. If the file returns 404, crawlers assume everything is allowed.
Does Crawl-delay work?
Some crawlers honour it and some ignore it. Check the documentation of the crawler you care about.