Crawl hygiene – Glossary

Optimizing how search engines crawl your site so they focus on important, indexable pages instead of wasting requests on low‑value URLs.

Crawl hygiene is the practice of keeping a site clean, efficient, and predictable for search engine crawlers. The goal is to ensure limited crawl budget is spent discovering and revisiting your highest‑value, indexable URLs—rather than being wasted on errors, duplicates, redirects, thin content, or parameter junk.

Why it matters

• Faster discovery and re‑crawling of new or updated content
• Fresher indexing and more stable rankings
• Reduced server load and fewer wasted requests
• Clearer signals about what should be canonical and indexable

Core checks

• Indexed vs indexable: Compare pages that could be indexed with pages that are actually indexed; large gaps indicate discoverability or quality issues.
• Crawl activity coverage: Review Googlebot crawl stats vs. total URL count to identify areas Google cannot or does not prioritize.
• Non‑indexables crawled: Minimize crawling of noindex, auth‑only, faceted/parameter URLs, and utilities.
• robots.txt disallows: Confirm blocks don’t prevent essential discovery of content or links.
• Low‑value pages indexed: Reduce indexation of thin or utility pages; guide crawlers with sitemaps and internal links.
• 4xx and 5xx errors: Fix broken or unstable endpoints; use 410 Gone for permanently removed content to stop re‑crawling.
• Redirect chains: Link directly to 200 targets; remove chains and loops to cut wasted requests.
• Canonicals: Limit duplication; align canonical tags with internal links and sitemaps. Note that canonicals don’t save crawl budget across duplicates on their own.

Implementation tips

• Keep sitemaps lean and accurate; include only canonical, indexable URLs.
• Use consistent URL design; avoid duplicate paths and uncontrolled parameters.
• Align internal linking with canonical targets; avoid linking to redirects.
• Gate low‑value tools or faceted pages behind robots.txt, <meta name="robots" content="noindex">, or both, where appropriate.
• Return proper status codes: 200 for canonical pages, 301 for permanent moves, 410 for removed content, and avoid soft 404s.
• Monitor crawl stats in Search Console and server logs; fix spikes in errors or redirect chains promptly.