Glossary

Crawl Budget

Crawl budget is the number of pages a search engine will crawl on your site in a given period.

Crawl budget is how many URLs search engines crawl on your site within a window, set by crawl capacity and crawl demand. It matters most for large sites, where crawling low-value URLs can slow indexing of important pages.

Example

Blocking faceted-navigation URLs frees crawl budget for a retailer's key category and product pages.