At the beginning of this month (May), I noticed that the EnviroGadget homepage had disappeared from the Google search results. Most of the other pages were still visible in the index, but mainly the home page plus a few other key pages were missing. At first I thought it was a penalty, but then I wasn’t doing anything to deserve one.

After some digging in Google Webmaster Tools, I noticed that there was an error with my sitemaps for most of my sites (which are currently on the same server). This was the error that I was getting:

URL timeout: robots.txt timeout. We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit.

After some research, I found an article “Banned” By Google? Find Out How to Entice Googlebot to Recrawl Your Site, which explained that the error was due to the Google bots being blocked in some way by the server, and therefore not being able to check the robots.txt for instructions on what to crawl. Essentially, blocking the Google bots was stopping my site from being crawled.

As a result of not being crawled, my site was removed from the index! Having asked my hosting company to remove the Google bots ip from the firewall rules, my sites started getting crawled again. And thankfully, they now appear in the search results once more.

Phew!