SpunkArt
Websites, APIs and search

How robots.txt works, with a file builder

robots.txt is a short text file at the root of a site that asks crawlers to stay away from certain paths. Well-behaved crawlers follow it. It is a request, not a lock.

Where it lives and how it is read

The file must be at exactly /robots.txt on each host. A file on the www host does not cover the bare domain or a subdomain; each needs its own. Crawlers read it before fetching other pages and cache it for a while.

The file is made of groups. Each group starts with one or more User-agent lines naming crawlers, followed by rules:

User-agent: *
Disallow: /drafts/
Disallow: /search
Allow: /drafts/press-kit.html

Sitemap: FULL-ADDRESS-OF-SITEMAP

How matching works

  • A crawler obeys the group whose User-agent matches its own name most specifically, and falls back to *.
  • Rules match from the start of the path. Disallow: /search blocks /search, /search?q=x and also /searchlight.html. Add a trailing slash when you mean a folder.
  • When an Allow and a Disallow both match, major crawlers pick the more specific rule, meaning the longer one.
  • * matches any characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks PDF files.
  • An empty Disallow: line means nothing is blocked.

robots.txt builder

The builder adds a leading slash to paths that lack one and drops duplicates. It runs in your browser only. Save the output as a plain text file named robots.txt.

What robots.txt cannot do

It does not hide anything. The file is public, so listing /secret-plans/ advertises that folder to anyone curious. Bad bots ignore it completely. And a blocked page can still appear in search results as a bare link if other sites link to it, because the crawler never reads the page to discover it should not be shown.

GoalRight tool
Save crawl effort on endless filter or search URLsrobots.txt Disallow
Keep a page out of search resultsnoindex meta tag or header, and do not block the page in robots.txt
Keep content privateA password or other access control on the server

Common mistakes

  • Shipping Disallow: / from a staging site to production, which asks every crawler to skip the whole site.
  • Blocking CSS and JavaScript folders, so search engines cannot render pages properly.
  • Combining Disallow and noindex on the same page; the crawler never sees the noindex.

Questions

Is a missing robots.txt a problem?

No. If the file returns 404, crawlers assume everything is allowed.

Does Crawl-delay work?

Some crawlers honour it and some ignore it. Check the documentation of the crawler you care about.