Googlebot link status code returns 200 when it should be 404

Question

I have set up some code to filter pages with certain words in the query string to redirect to a 404 or 410 page which works perfectly. The problem is when I check the logs, google bots are still showing the same link as 200. I have checked using different methods and the page is definitely coming up as a 404. I have also tested links through google and they respond with 404. I have supplied a short extract from my log as an example (changed IP and some text for privacy):

> 66.249.69.46     - W3SVC472 WIN9 80 GET /events/default.asp youth-zzz-yyy?___from_store=it&___store=dk&cat=54&price=1300- 200 0 14543 441 1 HTTP/1.1 .... (compatible;+Googlebot/2.1;++http://www.google.com/bot.html)

> 111.111.111.111  - W3SVC472 WIN9 80 GET /events/default.asp youth-zzz-yyy?___from_store=fr&___store=de&price=150- 404 0 1434 724 1 HTTP/1.1 Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:33.0)+Gecko/20100101+Firefox/33.0

In the first entry it appears to be from a google bot (IP confirms it is google) and the second is me clicking from a link in google. Both pages should show a response status of 404 but the first shows 200. Why?

This is my code in the default.asp which checks for a particular term in the the query string eg, zzz-yyy:

if (inStr(LCase(Request.QueryString), "zzz-yyy")) then 
    Response.Status = "404 Page not found"
    Response.End
end if

Simple and works but not for bots. Could it be cached somewhere? I tried to clear the domains cache but that did not help either.

Answer 1

I am posting this as the answer to my own question as after an exhaustive search I believe this is the answer I needed and may help others. When Google checks links which it does more regularly than indexing your site, it loads the cached pages, which in my case above return a 200 status as the cache copy still exists. So when I click on the links in Google search, it correctly loaded the 404 page as intended. But when I checked through Google webmaster tools using the "Fetch as Google" option, it loaded the cached page with a 200 response code. So I guess it's just a matter of being patient or using using Google tools to encourage it to re-index the site.