When running Sitemap Creator on the development version of cyotek.com, we found all links pointing to articles returned a 404 status code when crawling was attempted. But if same URL was copied into a browser, it would load correctly.
This surprised us, as cyotek.com is the main site we test Sitemap Creator and WebCopy on and they've always worked in the past. Next, we tried it directly on cyotek.com, and got the same result. However, this being the release version of the web, we started receiving error emails from the website (these are not sent from the debug builds).
The exception being reported was this:
This error message certainly raised eyebrows, as of course, this action does exist.
This is the current definition of the display article action:
As soon as we looked at the code, we realised what had happened.
By default both Sitemap Creator and WebCopy make
to obtain the headers for a given URL, such as the content type.
They use these headers to determine if they should go ahead and
download the entire file - Sitemap Creator won't download
anything that isn't
text/html for example.
And this is the problem - in the last update to cyotek.com, we
changed a few site settings to stop the number of error emails
occurring due to spammer activity. For some reason the
AcceptVerbs attribute was applied to the
Display action method
at this point. And as it is only set to accept
GET, it means our
HEAD calls automatically fail.
One changing the attribute, everything started working nicely again.
For once, a nice and simple mystery to solve, and a nice little tip which will hopefully help anyone else who has a similar issue.
- 2010-11-20 - First published
- 2020-11-21 - Updated formatting
Like what you're reading? Perhaps you like to buy us a coffee?