On the internet, no one knows you’re a dog… and the flip side is that I can claim to be a dog (or a Googlebot) whenever I feel like it.

These liars impersonate Googlebot in hopes that it will help them bypass bot controls; indeed, if you look up that IP in my log above, it comes from Virtual Machine Solutions LLC. This is a hosting provider, so someone (not Google) rented a server from them to send this traffic.

People do this, all the time, and it seems to have hit a new scale recently. Chris Siebenmann theorizes that “it’s a large scale campaign by a single abusive crawler by people who can afford to obtain a lot of servers at a lot of different hosting providers”, rather than a widespread rise in popularity for an old trick.

Lightning Q&A

Q: How do I verify that a request came from the real Googlebot?

A: Here are Google’s docs for verifying Googlebot; in short, you can spot-check with a couple of `host` commands or check against Google’s published IP ranges. Most major crawlers publish some kind of IP list, but the details vary; the JAFAR proposal aims to standardize these lists, though it’s still working its way through the standard-making process.

Q: What if I looked up the IP address and saw it was registered to Google LLC?

A: I’d still run through the verification process, since “Google LLC” is also the registration for Google Cloud hosting IP addresses. Though personally, I wasn’t able to find any recent examples of this - I wouldn’t be surprised if the Google Cloud anti-abuse team had some kind of controls to crack down on this.

Q: I checked and it really is real Googlebot. What should I do?

A: See Google’s docs on how to handle overcrawling of your site.

Q: You mean to tell me people can lie on the internet?

A: Yes :(

Q: What if we added another field to stop them from lying?

A: Please refer to RFC 3514, the “evil bit” standard. Unfortunately, bad actors do not comply with this standard.

On a more serious note, I’ve written at length about the Web Bot Auth HTTP Signatures proposal for my employer (as always, opinions on this blog my own). That proposal is a little more complicated than just another field since it involves a cryptographic signature, but I think it’s a reasonable path to “a user agent you can trust”.

*Q: You’ve convinced me some Googlebots are fake. But is it really “most”?

A: Okay, I concede that I don’t have data to prove this. I offer this weaker claim: most stories of disruptive Googlebot traffic (like the quotes at the start of this post) are caused by fake Googlebots. In my experience, the real Googlebot is very well-behaved; meanwhile, fake Googlebots are happy to hammer on a site until it explodes.

That being said, if you have seen disruptive behavior from Googlebots (real or fake), I’d like to hear about it - reach out at [email protected].