XML Sitemaps: Build a Reliable Content Discovery File

Decide what the file represents

A sitemap should express your intended public collection. It should not be an unfiltered export of every URL your application can produce. Content databases often contain drafts, preview routes, retired pages and several representations of the same article. Establish a publication rule before writing XML: the record is published, the route is public, the page is intended for search and the chosen address is its preferred destination. This turns sitemap generation into a content decision with a reproducible technical expression.

Start with the article records that produce visible pages. In an illustrative library with 120 published articles and 12 drafts, the article sitemap begins with the 120 published records. It does not gain value from listing the drafts or every filter combination. Add useful permanent category pages if they are intended to stand alone. A search results page with arbitrary query parameters belongs to a different class. Use your content inventory to identify these classes before deciding which ones qualify.

Keep address signals aligned

Use the same URL builder for the sitemap, internal article links and canonical metadata. A small difference such as an old hostname can turn a clean looking file into a list of redirects. Choose an HTTPS origin and a consistent trailing slash policy, then derive addresses from the stored slug. If your public domain changes, update that configuration deliberately. A production sitemap should never inherit a preview hostname merely because the build happened in a preview environment.

Google's sitemap guidance calls for fully qualified URLs and recommends listing the URLs you want in search results. That makes a sitemap a useful consistency check against canonical URLs. If an article declares another page as canonical, resolve the disagreement before submitting both. A sitemap entry is not a command that overrides the page's metadata, redirects or content. Multiple contradictory signals are a publishing defect to investigate, rather than an opportunity to submit more variations.

Make modification dates meaningful

Store publication and modification dates with the content record. The first describes when an article became public. The second should change when the article meaningfully changes. Updating an explanation after a product removes a feature is substantive. Rebuilding an unchanged page with a new navigation colour is usually not a reason to advertise fresh editorial content. Establish which edits update the modification field, including significant changes to sources, links or structured information, so editors and developers use the same rule.

The Sitemaps protocol makes lastmod optional and distinguishes the page's modification date from sitemap generation time. If a migrated article has no trustworthy modification history, omitting lastmod is more honest than assigning today's date to everything. An illustrative migration can retain known dates for 80 articles and omit uncertain dates for 40. That unevenness accurately describes the available information. The wider editorial judgement about substantial updates belongs in content freshness, especially when the visible updated date also changes.

Generate valid XML without fragile shortcuts

Use an XML serializer or a framework facility that handles encoding. URLs containing ampersands require XML escaping even when the URL itself is valid. A manually concatenated string can appear acceptable in a text editor while failing as a document. The root namespace, enclosing urlset element and loc entries must be correct. Validate the generated output, not only the template, because an unusual slug or imported URL may reveal problems that simple test records never expose.

Large collections need bounded files. The protocol limits a sitemap to 50,000 URLs and 50 MB uncompressed, so use a sitemap index when necessary. Smaller divisions can also make operational ownership clearer: separate stable articles from frequently changing product pages if different teams maintain them. Avoid splitting a small library into dozens of files without a reason. The aim is an understandable export that can be fetched and maintained, not an elaborate hierarchy that resembles a ranking strategy.

Check the file and its destinations separately

Fetch the public sitemap as an unauthenticated visitor. Confirm that it returns the XML you expect, rather than an HTML login form, a hosting error or a stale cached copy. Parse it and compare the resulting URL set with the intended published set. Duplicate entries, missing articles and unexpected hosts are straightforward defects. Make this comparison part of the publishing process so a removed record cannot remain in the sitemap indefinitely after its page disappears.

Then sample the listed destinations across templates and publication states. A sitemap can be perfectly valid while all its links return errors. Inspect the response status, robots directives and canonical destination for representative pages. Indexability checks explain why a reachable page may still be excluded. If the file lists an article that returns a permanent redirect, replace that entry with the final destination when appropriate. If it lists a deliberately excluded page, fix the generation rule rather than editing the XML by hand after each build.

Compare a crawler's discovered URL set with the sitemap as a separate diagnostic. URLs found through ordinary links but absent from the sitemap may reveal an overly restrictive generation rule. URLs in the sitemap but absent from a navigational crawl may reveal weak architecture. Neither difference automatically proves an error, because a crawler can miss routes for its own reasons. Investigate representative cases and classify them before changing the export. This comparison is especially useful after importing content from another system.

Keep the sitemap endpoint independent of optional editorial services where possible. If every request rebuilds the entire file by calling a slow external database, a temporary content service problem can make discovery information unavailable even while cached articles still work. A generated file published with the content release can be simpler for a stable library. A dynamic implementation can also work when its caching, failure handling and publication rules are deliberate. Choose the design that maintains an accurate public snapshot reliably.

Interpret submission feedback correctly

Make the sitemap discoverable through your robots.txt declaration or submit it using the search platform's supported interface. Submission creates a way to inspect processing and fetch problems. It does not establish that every destination deserves inclusion. When a platform reports it could not fetch the sitemap, investigate that request first. Rewriting article introductions will not repair an inaccessible XML endpoint. When the sitemap succeeds but a particular page remains absent, investigate the page and the platform's available diagnostics separately.

After an illustrative release adds ten articles, compare the live sitemap with the release record and follow a few new URLs. Confirm that the articles also have meaningful routes from category pages or related explanations. Orphan pages are still poor experiences even when their addresses appear in XML. Keep a small history of sitemap counts and unexpected changes, but interpret those counts as operational evidence. A healthy sitemap demonstrates a consistent discovery file; useful content and sound site architecture must do the rest.

Inspect a small valid XML example

This shortened example uses two real canonical article addresses from this library. A production generator should read its URLs and dates from the published content records rather than maintain a separate hand-written list. The date-time represents a substantive content revision. Omitting lastmod is preferable to guessing when a page changed.

Serve the XML from a stable public address with a successful response and an appropriate XML content type. Check that each listed address resolves directly to the intended public page and agrees with its canonical. A URL containing an ampersand must encode it as & in XML. The sitemap is a discovery document, so it should not contain account pages, internal search combinations or unpublished drafts just because the CMS knows those addresses.

To notify participating engines when a URL changes, use the separate IndexNow implementation guide. Keep the sitemap as the current inventory; a change notification does not replace it or establish that a page was indexed.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://thekingofaeo.com/learn/what-is-aeo</loc>
    <lastmod>2026-09-09T12:11:36.551Z</lastmod>
  </url>
  <url>
    <loc>https://thekingofaeo.com/learn/crawlability</loc>
    <lastmod>2026-09-09T12:11:36.551Z</lastmod>
  </url>
</urlset>

Sources and further reading

• Google sitemap guidanceSitemap URLs should be absolute; lastmod represents a significant page update; submission is a hint.
• Sitemaps XML protocolDefines required XML elements, UTF-8 encoding and escaping.