Splitting 404

404 error

I check the stats for my website on a daily basis. I split all the details down into a couple dozen reports. That sounds like overkill, but I only use those reports if the summary page points to an issue or anomaly.

Recently, I created an archive folder for posts over three years old. I knew this would cause some “file not found,” or technically, 404 errors. However, with my new sitemap xml files I was sure that the big search engines would have no trouble adjusting.

Boy, was I wrong. The smaller, and newer search engines had no problems at all. The new URLs were showing up after just a few days. The big boys though are still struggling three weeks later. I know my little website isn’t a priority, but I assumed this kind of stuff was automated. Again, I was wrong.

This all caused me to worry about my actual users. How were these 404 errors affecting them? The problem was that all 404 errors for my site went into a single report. So, I split the report in two. The first only showing errors from spiders and bots. The second showing actual users.

Fortunately, my users are like the little search engines. There was a period of three or four days of errors, rapidly falling off so that there were no errors of the last week. Now I feel better.

What confuses me most though is that the big boys come looking for the same files every day. These files are no longer listed in my sitemaps. Yet, one of the largest search engine on the planet tries to access a non-existent file every day for the last 20 days. When will it learn that the file is never coming back?

I fear that this a symptom of computer programmers not cleaning up after themselves. In one case, specifically WordPress’ bot Photon, It has been trying to access the same non-existent files ever since I stopped using WordPress over three years ago. I have even blocked them from accessing my site, but still they persist. Every day. For three years. This is one stupid bot.

Anyone, if my file changes caused problems for anyone, I’m sorry. Thanks.