TL;DR
Googlebot is Google's crawler used to automatically scan websites by following links from a page to another one and add them to the index based on the relevance.
What is Googlebot?
Googlebot, as the name says, is Google's crawler (also named bosts) in charge of crawling websites. Googlebot is the general name given for:
- A desktop crawler that simulates a user on the desktop (Googlebot Desktop).
- A mobile crawler that simulates a user on a mobile device (Googlebot Smartphone).
How does Googlebot access your site?
Googlebot's aim is to crawl and scan all the web sites and their pages via links: basically, Googlebot finds and reads all the new or updated content and adds it to the index based on its relevance.
The visits created by the Google crawlers may be displayed in the referrer logs, and they should be specified in robots.txt.
The Googlebot's goal is to crawl as many pages from a site as possible but to respect the disavow links selected by the website owner as well.
To ask Google to (re)index a website, please note these general guidelines:
- Crawling can take anywhere from a few days to a few weeks, but you can always monitor progress using either Google’s tools: Index Status report or the URL Inspection tool.
- Requesting a recrawl multiple times for the same URL or sitemap won't get it crawled any faster.
- The best way to verify that a request actually comes from Googlebot is to use a reverse DNS lookup on the source IP of the request.
- Google actively fights spammers; if you notice spam pages or sites in Google Search results, you can report spam to Google.
- If you want to prevent Googlebot from crawling content on your site, you have a number of options. Be aware of the difference between preventing Googlebot from crawling a page, preventing Googlebot from indexing a page, and preventing a page from being accessible at all by both crawlers or users.
