Robots.txt and Sitemaps: Two Files Worth Getting Right

Two files sitting at your site's root — robots.txt and your XML sitemap — do a disproportionate amount of quiet, unglamorous work in how search engines discover and prioritize your content. Both are easy to set up correctly and, unfortunately, just as easy to misconfigure without any obvious symptom until traffic mysteriously drops.

robots.txt: a request, not a lock

robots.txt tells well-behaved crawlers which parts of your site they're welcome to crawl. It's worth being precise about what it does and doesn't do: it's a voluntary convention, not an access control mechanism — it won't stop a page from being accessed directly, and in some cases a disallowed URL can still appear in search results (without a description, since the crawler was never allowed to read the page) if enough other signals point to it.

The single most damaging robots.txt mistake, and one that happens more often than you'd expect: a line like Disallow: / left over from a staging environment, accidentally deployed to production. That one line tells every well-behaved crawler to skip the entire site. It's worth checking your live robots.txt periodically, especially after any deployment or platform migration — this exact mistake has silently deindexed entire sites for weeks before anyone noticed the traffic drop.

Your sitemap: a map, not a mandate

An XML sitemap lists the URLs you consider worth crawling and indexing, optionally with hints about how often they change and how important they are relative to each other. It doesn't guarantee indexing — search engines still decide independently — but it's a strong signal, especially for pages that are hard to discover through normal internal linking, like content added after the last time your navigation was updated.

Two common sitemap mistakes: including URLs that redirect or 404 (stale entries left in after a page was removed or moved), and — the opposite problem — a sitemap that's stale in the other direction, missing entire sections of newer content because it's generated once and never regenerated as the site grows.

The two files should agree with each other

It's worth explicitly checking that you're not disallowing, in robots.txt, pages you're simultaneously listing in your sitemap as important. That contradiction happens more often than you'd expect on sites where the two files were set up at different times, by different people, without either one being cross-checked against the other.