A 60-Second Soft 404 Test (We Missed Ours for Weeks)

On September 19 we found that every URL that doesn’t exist on this site was returning HTTP 200 and a copy of the homepage, instead of a 404. We had already run Screaming Frog’s free crawler and Semrush’s free site audit on the site, weeks earlier, and neither said a word about it. This page is the test that finds the problem, what happened on our site, and what we changed. The tests below are ones you can run yourself.

The test (about a minute)

You are asking your own site for a page that shouldn’t exist. Pick a made-up path nobody would ever link to:

curl -s -o /dev/null -w "%{http_code}\n" https://your-site.example/this-page-does-not-exist-xyz-123

You want 404. Anything else is worth a closer look, and 200 is the classic soft 404: the server says “success” for a page that isn’t there, whether it shows visitors an error message or, as ours did, just some other page. Google reports these as “soft 404” errors in Search Console’s page indexing report (Google’s help page for that report).

A status code alone can hide the worst version of this, where the server hands back your homepage for everything. So compare what came back:

curl -s https://your-site.example/this-page-does-not-exist-xyz-123 | grep -o "<title>[^<]*</title>"
curl -s https://your-site.example/ | grep -o "<title>[^<]*</title>"

If the two titles are identical, that’s the homepage answering for every address. Two variants are worth adding, because they’re what a mistyped link looks like in the wild: the same made-up path with a trailing slash, and a real page with the wrong extension (for example /about.htm when the page is /about/).

What happened on this site

• Around August 21 and 22: we ran Screaming Frog’s free crawler and Semrush’s free site audit against the site while testing them for our Screaming Frog and Semrush reviews (published August 21 and 22; Semrush’s own screenshot dates its audit August 22). Neither reported that unknown URLs returned 200.
• September 19: while trying to work out why Cloudflare’s bot dashboard showed a jump in unsuccessful (4xx) requests, we started requesting made-up paths ourselves. /wp-login.php, /llms-full.txt and /a/b/c/d.html all came back 200 with the identical 4,844-byte page: our homepage. We wrote those results down that day; we did not save a raw command transcript, so treat the byte count as our note, not a recording.
• The cause: the site had no 404.html. Cloudflare Pages’ documentation says that without a top-level 404.html, Pages assumes you are deploying a single-page application, and its default behavior for that case matches incoming paths to the root (Cloudflare’s “Serving Pages” documentation). For unknown paths, what we observed was a 200 with the homepage.

Why the crawlers missed it

A crawler asks for URLs it has discovered, through links and the sitemap. Every one of those URLs exists, so nobody ever asks the question “what does this site do with an address that doesn’t exist?” That’s our explanation of the mechanism, not something either vendor has told us.

We can’t say the same about Ahrefs’ Site Audit. Its first crawl of the site happened after the fix, so we have no evidence either way about what it would have done.

The fix

One small file, an ordinary Astro page at src/pages/404.astro:

---
import BaseLayout from '../layouts/BaseLayout.astro';
---
<BaseLayout title="Page not found — Solo Stack Lab" description="This page doesn't exist. ...">
  <h1>Page not found</h1>
  <p>There's no page at this address. ...</p>
  <ul>
    <li><a href="/reviews/">All firsthand reviews</a></li>
    <li><a href="/">Home</a></li>
  </ul>
</BaseLayout>

The build turns it into 404.html, and Cloudflare Pages serves that file for unknown paths. Our sitemap plugin leaves it out of the sitemap (we checked: zero mentions of it). Re-running the September 19 probes on September 25:

The made-up path from the test above (/this-page-does-not-exist-xyz-123) now returns 404 as well, and its title is “Page not found — Solo Stack Lab” instead of the homepage’s title, which is the second half of the test. We don’t have a recorded “before” for that exact path; our September 19 note covers the three paths above and says any unknown path behaved the same way.

Make it a standing check

We didn’t want to depend on remembering to look. A reader on Indie Hackers suggested asserting on status codes for the requests a broken link or a wrong config would make, instead of running a full crawl. We built that into a script we can run after any deploy (we haven’t wired it into an automatic step yet). It checks that:

• nonsense paths, trailing-slash variants and wrong file extensions return a real 404, and that the 404 body isn’t the homepage;
• http://, www and missing-trailing-slash addresses all end up at the one canonical URL, in a single hop;
• every URL in the sitemap returns 200 and points to itself as its canonical page;
• a browser and a Googlebot-style user agent get the same status codes (a spoofed user agent from your own machine isn’t a real crawler, so this only catches bugs that depend on the user agent).

To make sure it could fail, we pointed it at a throwaway server built to reproduce both of our September bugs. It failed 12 of the 14 checks that apply to a non-production address (the http and www checks only run against the real domain), and it passes 17 of 17 against the real site.

What we don’t know

• Whether Google ever indexed a junk URL. The Search Console indexing report we looked at on September 25 (its data was last updated September 21) listed no “soft 404” reason, only redirects and pages Google hadn’t crawled or indexed yet. That report only covers URLs Google chose to crawl.
• What the unsuccessful bot requests in Cloudflare were asking for. The dashboard showed the count, not the paths, so we still can’t say those requests hit the soft 404 rather than something else.
• Any effect on rankings or traffic, in either direction. The site is new and small, and we didn’t measure one.

For a related trap in the same family, see how to sanity-check your first backlink report, which covers the www mistake we also found that week.

Tools mentioned in this guide