Technical SEO Beyond the Basics: Crawl Budget and Indexing

For a small site, everything gets crawled and the technical SEO story ends at the basics. For a larger site — thousands of pages, filters, a store, user-generated content — a new problem appears: the important pages are not being crawled often enough, or not indexed at all.

What crawl budget actually is

Search engines allocate a limited amount of crawling to each site, based on how fast the site responds and how valuable the content seems. "Crawl budget" is shorthand for that limit. It only matters at scale — but when it matters, wasting it on low-value URLs means your real pages get visited less.

Common ways sites waste it

Faceted navigation generating endless filter-combination URLs.
Session IDs, tracking parameters and sort orders creating many URLs for the same content.
Infinite calendars, paginated archives with no end, and internal search result pages left crawlable.
Large numbers of redirects and soft-404s that the crawler still has to process.

How to diagnose it

Search Console coverage report — look at "Crawled - currently not indexed" and "Discovered - currently not indexed". Large numbers here mean the site is producing pages search engines do not think are worth keeping.
Server log files — the only fully accurate record of what search engines actually crawl. They show which sections get attention and which are ignored.
Index coverage vs. sitemap — if your sitemap has 5,000 pages and 1,500 are indexed, find out which type is missing.

The fixes

Block low-value URL patterns in robots.txt or with noindex, depending on whether you want them crawled at all.
Canonicalise parameter variants to the clean URL.
Keep the sitemap limited to canonical, indexable pages, and keep it accurate.
Improve the pages that are "crawled but not indexed" — usually they are thin or duplicative.
Speed the site up; a faster server earns more crawling.

In short

At scale, technical SEO becomes about not wasting the crawler's attention: prune low-value URLs, canonicalise duplicates, keep the sitemap honest, and strengthen thin pages. If your site is large and pages are missing from the index, send us the domain and we will diagnose where the budget is going as part of an SEO engagement.