搜索引擎与爬虫

googlebot user agent

aurora

Usually, the User-agent of Googlebot looks like this:

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

This is the User-agent for the desktop version of the Google search engine. However, Googlebot has other User-agents as well. By visiting the Googlebot page, we can see that there are many different User-agents.

For example:

  • Googlebot-Image/1.0: This is the crawler agent for the image search engine.

  • Googlebot-Video/1.0: This is the crawler agent for the video search engine.

  • SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html): This is the crawler agent for the mobile version of the search engine.

  • Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.5672.126 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html): This version has many variations, and almost every new version update of the Chrome browser changes the Chrome version number in this User-agent.

There are also other Google product crawler user agents, such as Google Adsense, Google Ads, etc., which belong to Google's crawler programs.

In conclusion, it can be challenging to determine whether it is a genuine Googlebot crawler through User-agent checking due to the following reasons:

User-agents can be forged.

Google has a wide range of products, making it difficult to determine which Google product a User-agent belongs to.

Google updates its User-agents quickly, often with Chrome version updates, resulting in new User-agent information.

Therefore, we can only rely on specialized tools to detect whether it is a genuine Google crawler.