The launch-readiness audit

robots.txt & sitemap.xml: letting search engines in

No robots.txt or sitemap means crawlers guess at your site and pages get missed. Learn what Actuant checks, the one line that accidentally blocks all crawlers, and how to fix indexing.

Last updated 2026-07-19

This weight-8 check fetches your /robots.txt and /sitemap.xml and confirms crawlers are both allowed in and pointed at your pages.

What Actuant checks

  • /robots.txt responds, and does not disallow the whole site (Disallow: /).
  • /sitemap.xml responds and contains a real <urlset> or <sitemapindex>.
  • Both present → pass. One present → partial. A Disallow: / that blocks every crawler → fail ("you're invisible on purpose").

A separate Indexable check (weight 4) catches the other way to hide: a <meta name="robots" content="noindex"> left on the page.

The one line that hides you

The most common self-inflicted wound: a leftover Disallow: / from a staging config, shipped to production. It tells every search engine to skip your entire site. Actuant flags this explicitly.

Why it matters

Without a sitemap, crawlers discover pages only by following links — slow, and easy to miss deep pages. Without a sane robots.txt, you're relying on defaults. Together they cut indexing time from weeks to days.

How to fix it

  1. 1

    Serve robots.txt

    Allow crawlers and point at your sitemap. In Next.js, add a robots.ts that returns rules; most builders have a toggle.

  2. 2

    Serve sitemap.xml

    List your real URLs. Next.js generates one from a sitemap.ts; builders usually auto-generate.

  3. 3

    Submit in Search Console

    Add the site to Google Search Console and submit the sitemap URL to kick off indexing.

robots.txttxt
User-agent: *
Allow: /

Sitemap: https://yourapp.com/sitemap.xml