What is an Ahrefs bot?
AhrefsBot is a web crawler used by Ahrefs. Ahrefs is one of the leaders in the SEO tools market. Ahrefs bot is a commercial bot collecting data for the Ahrefs SEO suite. On June 3, 2022, Ahrefs founded a search engine, Yep. Since then, Ahrefs bot has collected data for commercial use in the SEO suite and search engine purposes.
Ahrefs bot collects several data types, like:
- outbound and inbound website links,
- link anchors,
- content and others,
and adds them to its database. It also checks website resources such as CSS, JS, and images.
If needed, it is possible to block the AhrefsBot user agent by adding it to your website’s excluded bot list in the robots.txt file. Blocking the bot will prevent your website from crawling and indexing its pages.
What is the size of the Ahrefs database?
Ahrefs has one of the most significant link databases in the world. Form Ahrefs statistics (June 2023), the database has:
- 13.5 billion pages
- 22.1 billion total keyword index size
- 35.0 trillion backlink index
- organic search traffic index from 189 locations includes 596 million keywords
Ahrefs crawls 5 million pages every minute and discovers over 10 million pages every 24 hours. More info about this data is here https://ahrefs.com/big-data.