Insights › Technical SEO
robots.txt: the rules, the mistakes, and the one-line check
robots.txt tells crawlers which pages to skip. A misconfigured one blocks your entire site from being indexed — and it's a one-line mistake. Here's what to put in it, what to keep out, and how to check it in 10 seconds.
Aug 8, 2026 · 5 min read
The short version
robots.txt is a text file at the root of your domain that tells well-behaved crawlers which URLs to leave alone. The most common launch mistake is a generated deploy that ships with Disallow: / — which tells every crawler to index nothing. One line, total indexing blocker.
What a correct robots.txt looks like
For most launched apps, the file should allow everything except admin routes and internal API paths. The sitemap reference at the bottom is the discovery chain — without it, a new site with no inbound links can stay undiscovered.
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Sitemap: https://yourapp.com/sitemap.xmlThe rules
- `Disallow: /` blocks everything. If this line is present and unqualified, no crawler will index any page. This is the right setting for staging; it's a launch-blocker for production.
- `Allow: /` with specific disallows is the production default. Allow everything, then carve out the paths that shouldn't be indexed — admin panels, internal APIs, search pages.
- The `Sitemap:` directive is the discovery chain. It tells crawlers where your sitemap lives. Without it, they may never find the sitemap on their own.
- robots.txt is a request, not a command. Malicious crawlers ignore it. Legitimate ones (Google, Bing) respect it. Use
noindexmeta tags for stronger protection on pages that must not appear in results. - A missing robots.txt is fine. An empty 200 or a 404 on
/robots.txtmeans "no restrictions." That's better than a restrictive one you didn't mean.
The one-line check
Open https://yourapp.com/robots.txt in a browser. If you see Disallow: / on its own line, your site is invisible to every search engine. Remove it.
The second check: confirm the Sitemap: line points to a URL that actually returns XML. If the sitemap 404s, Google can't discover your pages.
Common mistakes in generated apps
| What's in the file | What it does | The fix |
|---|---|---|
Disallow: / (production) | Blocks every crawler from indexing any page | Replace with Allow: / |
No Sitemap: line | Google may never find your sitemap | Add Sitemap: https://yourapp.com/sitemap.xml |
Disallow: / but pages have noindex | Contradicts the noindex — inconsistent signals | Pick one: robots.txt or meta noindex, not both |
| Blocks CSS/JS assets | Google can't render the page for its second-pass JS rendering | Allow .css and .js in robots.txt |
| Returns 500 or a redirect | Crawlers can't read it; defaults to no restrictions, but it's a loose end | Serve a plain 200 with the correct directives |
Actuant checks your robots.txt for the presence of Disallow: /, verifies the Sitemap: reference resolves, and flags any conflict with your noindex tags.
Keep reading
sitemap.xml: what goes in it, what breaks it, and how to tell it's working
A sitemap tells search engines which URLs to crawl. Get it wrong and Google wastes its crawl budget on dead pages or misses your content entirely. Here's what belongs in it, how generated apps break it, and how to verify it.
Canonical tags: the rules that actually matter
A canonical tag tells Google which URL is the real one when a page is reachable at several addresses. Get it wrong and you split your ranking signals or deindex the wrong page. Here are the seven rules, and how generated apps break them.