SpunkArt
Websites, APIs and search

XML sitemaps: what goes in one and how to make it

An XML sitemap is a list of the pages you want search engines to know about. It does not force anything to be indexed, but it helps crawlers find pages that your links alone might hide.

What the file looks like

A sitemap is a plain XML file, usually saved as sitemap.xml at the root of the site. It has one <urlset> element, which carries the standard sitemap namespace, and one <url> entry per page:

<url>
  <loc>FULL-ADDRESS-OF-PAGE</loc>
  <lastmod>2024-05-02</lastmod>
</url>

loc is required and must be a full, absolute address including the scheme. lastmod is optional and should change only when the page content really changes. The older priority and changefreq fields are widely ignored, so the builder leaves them out.

Sitemap builder

The builder removes blank and duplicate lines, adds a leading slash where missing, percent-encodes spaces and escapes characters such as & that are not allowed raw in XML. It runs entirely in your browser.

Which pages to include

  • Include pages that return a 200 status, are indexable and are the canonical version of their content.
  • Leave out redirects, 404 pages, pages marked noindex, duplicates with tracking parameters, internal search results, and login or cart pages.

Every entry that fails these rules sends a mixed signal: you list it as important while another tag says ignore it.

Limits and splitting

One sitemap file may hold up to 50,000 URLs and must not exceed 50 MB uncompressed. Larger sites split pages across several files and list those files in a sitemap index, which uses <sitemapindex> and <sitemap> elements in place of <urlset> and <url>. Splitting by section, such as one file for articles and one for products, also makes problems easier to find in reports.

Telling search engines about it

  1. Upload the file and open it in a browser to confirm it loads.
  2. Add a line Sitemap: followed by its full address to your robots.txt.
  3. Submit it in each search engine’s webmaster tool if you use one, and check the report for errors.

Common mistakes

  • Mixing the www and bare domain, or the two schemes, inside one sitemap.
  • Setting every lastmod to today on each build, which teaches crawlers to ignore the field.
  • Forgetting to regenerate the file after adding or deleting pages.

Questions

Does a small site need a sitemap?

If every page is linked from the navigation, crawlers will find them anyway. A sitemap still costs little and helps you spot pages you forgot.

Can I list pages from another domain?

Normally a sitemap lists pages on the same host where it lives. Cross-site listing needs extra verification, so keep each site’s pages in its own file.