No robots.txt or sitemap.xml in production
why is my site not showing up on Google even though it's live Usually because /robots.txt or /sitemap.xml return a 404, still point at a placeholder domain, or were never wired into the deploy. Search engines fall back to slow, incomplete crawling without them. Confirm with `curl -sI https://example.com/robots.txt` and `curl -sI https://example.com/sitemap.xml`, then generate both from your actual route list and redeploy.
Seen in 37 of 450 scanned projects (8%). Based on automated scans of 450 production deployments across solo and small-team SaaS projects.
How to tell you have it
- curl /robots.txt or /sitemap.xml returns 404 on the live domain
- Search Console shows 'Sitemap could not be read' or zero discovered URLs
- robots.txt still references a staging or fork's original domain
- New pages take weeks to get indexed with no crawl errors reported anywhere else
- A login or admin route shows up in Google search results
Why it matters
Crawlers do not guess your site structure, they read robots.txt for crawl rules and sitemap.xml for a URL manifest. When both are missing, Googlebot and Bingbot fall back to following internal links from whatever pages they already know, which is slow for anything not tightly interlinked and can miss entire sections of a site for months.
A robots.txt or sitemap that survived a fork or a domain migration but still points at the old domain is worse than having none at all. Search engines read the sitemap's own URLs as canonical hints, so a sitemap full of another host's URLs actively confuses indexing rather than helping it, and any automated ping to search engines silently fails.
A sitemap missing lastmod on most entries removes the one signal that tells a crawler which pages changed and deserve a recrawl. Crawl budget on anything but the largest sites is finite, and without lastmod, engines treat every URL as equally stale, which slows how fast content updates actually get reflected in search results.
A publicly reachable login or admin page with no noindex directive and no robots.txt disallow rule can end up indexed and shown in search results, which leaks internal tooling to anyone searching your product name and looks unprofessional to prospects who find it before your marketing pages.
How Heygents detects it
Deep Scan runs `curl -sI` and `curl -s` against the live domain's `/robots.txt` and `/sitemap.xml`, checks the response status and content type, parses the sitemap for URL count, lastmod coverage, and whether hostnames match the live domain, and reads the rendered HTML of a sample of pages for canonical tags and noindex meta on sensitive routes.
How to fix it
- Confirm both files are actually missing or broken in production Do not trust what is in the repo, check what is actually being served. A file can exist in the codebase and still 404 in production if it never got deployed or is excluded by a build step.
- Generate robots.txt from the real route list, not a hand-typed guess Hand-written robots.txt files drift from the actual route list within a few deploys. Point crawlers at the sitemap and disallow only routes that genuinely should not be indexed, such as auth and internal API paths.
- Generate sitemap.xml with lastmod from your actual content source Pull the URL list and last-modified timestamp from your CMS, database, or file mtimes at build time rather than maintaining a static file by hand. This is the script equivalent of the failure mode above where entries have no lastmod at all.
- Fix a leftover placeholder or wrong-domain sitemap after a fork If the project was cloned or forked from a starter kit, grep the generated output for any domain other than the real production one before you redeploy.
- Serve canonical URLs over https to match the live protocol A canonical tag emitted over http on a site that is actually https tells engines the http version is authoritative, which can suppress the https page from search results entirely. Set the canonical from the request's actual scheme, do not hardcode http.
- Add noindex to any publicly reachable login or admin route Disallowing a path in robots.txt only stops crawling, it does not remove a URL that is already indexed or linked from elsewhere. Add a noindex meta tag directly on the page so engines that already found it will drop it.
- Re-verify after every deploy, not just after the fix is written This is the most common way this check reappears: the fix gets merged, but the build step that generates the files is skipped, cached, or excluded from the production bundle. Re-run the same curl checks against the live domain after each deploy, not against localhost or a preview URL.
Confirm both files are actually missing or broken in production
curl -sI https://example.com/robots.txt
curl -sI https://example.com/sitemap.xml
curl -s https://example.com/sitemap.xml | head -20Generate robots.txt from the real route list, not a hand-typed guess
User-agent: *
Disallow: /login
Disallow: /admin
Disallow: /api/
Allow: /
Sitemap: https://example.com/sitemap.xmlGenerate sitemap.xml with lastmod from your actual content source
const { execSync } = require('child_process');
const routes = getPublishedRoutes(); // from your DB or content source
const urls = routes.map(r => `
<url>
<loc>https://example.com${encodeURI(r.path)}</loc>
<lastmod>${r.updatedAt.toISOString().split('T')[0]}</lastmod>
</url>`).join('');
const xml = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">${urls}
</urlset>`;
require('fs').writeFileSync('public/sitemap.xml', xml);Fix a leftover placeholder or wrong-domain sitemap after a fork
curl -s https://example.com/sitemap.xml | grep -oE 'https?://[^/<]+' | sort -uAdd noindex to any publicly reachable login or admin route
// Emit a real noindex on routes that must never rank.
// robots.txt Disallow only stops crawling - it does not deindex a known URL.
export const metadata = {
robots: { index: false, follow: false },
};
// Plain HTML equivalent, rendered into <head>:
// <meta name="robots" content="noindex, nofollow">Re-verify after every deploy, not just after the fix is written
curl -sI https://example.com/robots.txt | grep -i '^HTTP'
curl -s https://example.com/sitemap.xml | grep -c '<loc>'Find this in your own projects, automatically
Heygents runs a read-only Deep Scan across every project you own, finds issues like this one, and hands you a ready-to-run fix an AI agent can execute and verify. A solo developer gets the audit, the backlog and the fix loop in one place.
Open Heygents →Frequently asked questions
Will adding robots.txt and sitemap.xml immediately get my site indexed?
No, they only make crawling more efficient and reliable, they do not force indexing. Submit the sitemap in Search Console and Bing Webmaster Tools after adding it so engines are notified directly instead of waiting to discover it during a routine crawl of your domain.
Do I need a sitemap if my site only has a handful of pages?
It still helps, but the bigger risk on small sites is usually a missing or wrong robots.txt disallow rule on internal routes. For sites under roughly 50 pages that are well interlinked, prioritize fixing robots.txt and canonical tags first, then add the sitemap as a fast follow.
Why did my sitemap disappear after I migrated to a new framework?
Static file generation for robots.txt and sitemap.xml is often framework-specific, a Next.js `public/sitemap.xml` or a route handler does not carry over automatically when you switch frameworks. Check the new framework's static asset or route conventions and regenerate both files as part of the migration checklist, not as an afterthought.
Should unencoded non-ASCII characters in sitemap URLs matter?
Yes, sitemap URLs must be percent-encoded per the XML sitemap spec. A raw non-ASCII path segment, such as an accented character or a Hebrew slug, can cause strict sitemap parsers to reject the entire file rather than skip the one bad entry, so encode every `loc` value with your language's URI encoder.
Do I need an RSS feed as well as a sitemap?
Only if you publish on a regular cadence, such as a blog or changelog. RSS is not an indexing signal for Google the way a sitemap is, but it lets syndication tools, email digests, and some AI crawlers pick up new content faster than waiting for a recrawl, so it is a cheap addition once publishing is automated.