The launch-readiness audit
robots.txt & sitemap.xml: letting search engines in
No robots.txt or sitemap means crawlers guess at your site and pages get missed. Learn what Actuant checks, the one line that accidentally blocks all crawlers, and how to fix indexing.
Last updated 2026-07-19
This weight-8 check fetches your /robots.txt and /sitemap.xml and confirms crawlers are both allowed in and pointed at your pages.
What Actuant checks
/robots.txtresponds, and does not disallow the whole site (Disallow: /)./sitemap.xmlresponds and contains a real<urlset>or<sitemapindex>.- Both present → pass. One present → partial. A
Disallow: /that blocks every crawler → fail ("you're invisible on purpose").
A separate Indexable check (weight 4) catches the other way to hide: a <meta name="robots" content="noindex"> left on the page.
The one line that hides you
The most common self-inflicted wound: a leftover Disallow: / from a staging config, shipped to production. It tells every search engine to skip your entire site. Actuant flags this explicitly.
Why it matters
Without a sitemap, crawlers discover pages only by following links — slow, and easy to miss deep pages. Without a sane robots.txt, you're relying on defaults. Together they cut indexing time from weeks to days.
How to fix it
- 1
Serve robots.txt
Allow crawlers and point at your sitemap. In Next.js, add a
robots.tsthat returns rules; most builders have a toggle. - 2
Serve sitemap.xml
List your real URLs. Next.js generates one from a
sitemap.ts; builders usually auto-generate. - 3
Submit in Search Console
Add the site to Google Search Console and submit the sitemap URL to kick off indexing.
User-agent: *
Allow: /
Sitemap: https://yourapp.com/sitemap.xmlKeep reading
AI-search visibility
AI answers from ChatGPT, Claude, and Perplexity are where discovery is moving. An llms.txt file is how they read your site accurately. Learn what Actuant checks and how to write one.
Title & description
A missing <title> or meta description means Google writes your search result for you — badly. Learn what Actuant checks, how long each should be, and how to set them in any builder.
SEO for AI apps
AI builders ship JavaScript SPAs that look empty to crawlers. Learn why your Lovable or Bolt app may be invisible to search and AI engines, and the fixes that make it indexable.