What ever Google giveth, Google can take away. That’s t he case for Google Offers. If you did a search for
Notice the screen shot for the search “offers” found currently in Google:
Although Google Offers is on the top of the results, the snippet reads, “A description for this result is not available because of this site’s robots.txt – learn more.” Because you get a page that would return a 404 status code when you click through, shouldn’t that mean that Google should remove the page from the search results?
Here is what the Google Offers page looks like:
This is where old links and a site’s reputation (after all, Google has a pretty big reputation) can continue to keep content showing up after any and all content has been changed or deleted from the page. It’s funny how even with the page being blocked by the robots.txt, Google’s crawlers are still trying to access the URL to see if the disavaow and 404 status has changed.