Finding the 404s?

[#1]

I have my <username>.co.uk domain and web site hosted at Plusnet on which webstats tells me I'm getting a significant number of 404 errors. Is there anyway of finding what these are? Webstats doesn't seem to offer any options to do this. Do I need to request raw logs?

[#2]

If your website is indexed by search engines they will look for /robots.txt. If that doesn't exist 404s.will result. Also googlebot looks for a number of "well known files" associated with other registrations.which result in 404s if they don't exist.

However I think you'd need raw logs to get the details.

[#3]

Thanks for the reply. I've certainly got a robots.txt in the root so it's not that but could be other files, as you say. Also, there may be some deep links missing here and there as I've moved most of the site to <username>.me.uk which is hosted elsewhere and put in redirectors to take visitors there.

I've opened a ticket to ask for raw logs and will post again when I have something to look at.

[#4]

@Snapshot - If you’re getting a lot of 404 errors most of these can possibly be attributed to the lack of a favicon. This is an icon that appears in the browsers tab, for Plusnet it’s an outlined cross and for Youtube it’s a red play button, you get the idea. However, if you have a favicon then more investigation will be needed using the log files.

[#5]

Thanks for the suggestion. I don't have a favicon so I'm looking to see if I can add one to my site. It'll be an enhancement to the site even if it's not the cause of the problem.

[#6]

I host a few websites on my own server. From my experience a lot of the 404s are bots looking for files which contain known exploits - e.g. old unpatched versions of Wordpress, Joomla etc. If they actually checked what, if any, CMS system a site used they could optimize their attacks somewhat...

[#7]

@Snapshot - Are you any the wiser as to the source of your 404s?

[#8]

No, no reply to my ticket yet.