Technical SEO audit: How to check and fix crawl errors

Technical SEO audit: how to check and fix crawl errors, meaning everything that prevents Google from accessing, understanding and correctly indexing pages: blocks in robots.txt, server errors, redirect chains, incorrect canonicals, inaccessible resources, duplicate URLs. These are invisible to visitors and can keep half the site out of the index for months. This guide describes the procedure we apply to every new site, by area, with the tools, common errors and prioritization of fixes by impact.

A technical audit does not bring traffic on its own; it is the foundation: good content and links achieve nothing on pages Google cannot see. That is why we do it before any content work.

Technical SEO audit: how to check and fix crawl errors

The tools

• Search Console: the page indexing report, crawl stats, URL inspection, Core Web Vitals; the source of truth, free.
• Your own crawler (Screaming Frog, free for up to 500 URLs; Sitebulb; or the GOAI site analyzer for basic checks): titles, descriptions, codes, canonicals, links, depth.
• curl from the terminal, for headers, redirects and responses by user agent.
• Server logs, for what Googlebot actually does.
• PageSpeed Insights and Lighthouse, for speed and rendering.
• The structured data validator and header scanner, for markup and security.

Area 1: access

1. robots.txt: exists, is static, does not block CSS, JavaScript, images or content sections; wildcard rules are checked in the robots.txt report in Search Console. A Disallow: / left over from development is the most expensive mistake.
2. Firewall and protection services: Googlebot (verified by IP address) receives 200, not 403 or verification pages; the live test in Search Console shows what it received. We have repeatedly encountered protection services that served bots a verification screen with a 200 code, indexed as a "page".
3. Server speed: average response time in crawl stats below 300 ms; 5xx errors at zero; crawl capacity depends on these.
4. DNS and certificate: no host errors in crawl stats; a valid certificate for all names.

Area 2: indexing

1. The page indexing report, filtered to submitted pages: every reason for non-indexing with its list; interpretation is in the guide to coverage reports.
2. Unintentional noindex: the crawler lists pages with meta robots noindex or an X-Robots-Tag header; compare with what you want indexed.
3. Canonicals: each indexable page with a self-referencing canonical; variations (parameters, filters) pointing to the main page; no canonical pointing to the homepage or to http.
4. Duplicates: URL variations (slash, uppercase, www, http), parameters, pagination, print versions; fixes are in the guide to duplicate content.
5. Thin pages and soft 404s: pages with fewer than 100 words, empty categories, search results; noindex or consolidation.
6. Hreflang, for multilingual sites: reciprocal, with correct codes and a self-referencing canonical on each version.

Area 3: URLs and redirects

1. A single canonical version of the domain, with a direct redirect (one hop) from all others; curl -IL on each variation.
2. Chains and loops: the crawler lists them; each chain is shortened to one hop.
3. Internal links to redirects and 404s: rewritten to the final destination; on our blog, consolidating 13 pairs of articles required rewriting 179 internal links on the same day.
4. Old URLs after migrations: each with a 301 to its new equivalent; those without an equivalent, 410.
5. URL structure: short, with words, without parameters in internal links.

Area 4: rendering and resources

1. The rendered HTML from URL inspection contains the main content, headings and links; anything appearing only after late-executed JavaScript or after interaction is not seen.
2. Resources (CSS, JavaScript, images, fonts) accessible to Googlebot, from domains that do not block it.
3. Heavy pages: under 2-3 MB and fewer than 100 requests; appropriately sized images, with srcset.
4. Core Web Vitals on mobile, from real data: LCP below 2,5 seconds, INP below 200 milliseconds, CLS below 0,1; fixes are in the article on Core Web Vitals optimization.

Area 5: sitemap, structure and internal links

1. XML sitemap: only canonical, indexable 200 URLs; real lastmod; submitted and with Success status; the rules are in the guide to submitting an XML sitemap.
2. Depth: important pages no more than three clicks from the homepage; the crawler reports depth.
3. Orphan pages: in the sitemap but without any internal links; link to them or remove them.
4. Menu and footer: complete on mobile, without links to redirects.
5. Anchors: descriptive, not "here".

Area 6: logs

The access log filtered for verified Googlebot shows what Google actually crawls: the proportion of requests to useful pages versus parameters and useless URLs, response codes, frequency by section. On large sites, it is the only source for crawl budget; the method is in the article on crawl budget, and reading logs in cPanel is covered in the guide to error and access logs.

The errors we find most often

Error	Frequency in audits	Effect	Fix
Internal links to redirects	almost all sites	wasted budget, diluted signals	rewrite to destination
Duplicate URLs (slash, www, parameters)	most	incorrect canonicals selected	redirects and canonical
Accidental noindex or blocking	one in five	entire pages missing from the index	correction and indexing request
Sitemap with dead or redirected URLs	half	warnings, wasted budget	regenerate from current data
Bots blocked by protection	one in ten, increasing	indexing stopped or incorrect content	exceptions for verified bots
Test site indexed	one in ten	duplicate of production	password, then removal
LCP above 4 seconds on mobile	half	poor experience, reduced crawling	images, cache, scripts

A technical audit, through an example

A store with 3.000 products and organic traffic slowly declining for a year. Access: correct robots.txt, but the protection service blocked Googlebot on category pages with filters (403 in logs, with the "too many requests" pattern). Indexing: 28.000 known URLs, 2.600 indexed; 19.000 duplicates without a canonical (the product in three categories, plus sorting). URLs: two chained redirects for http and www; 1.200 internal links to redirects. Rendering: prices and availability fetched through JavaScript after loading, invisible in the rendered HTML for half the products. Sitemap: 4.000 URLs, of which 900 redirected. Prioritization: the Googlebot exception in protection (day 1), canonicals and noindex on sorting (week 1), one-hop redirect and link rewriting (week 2), prices in HTML (month 1), regenerated sitemap (week 1). After two months, indexed URLs had fallen to 3.400 (products and categories), while product impressions increased by 45%; no new content.

Custom tools for repeated checks

A 30-line script with curl, run weekly from cron, checks the list of main pages for: response code, presence of meta robots noindex, canonical, Content-Type header, response time, and sends email only when deviations occur. A second script compares the sitemap with the actual responses of its URLs, monthly. On the sites we manage, these two scripts caught a noindex introduced after a module update and a redirect loop after a DNS change in the same week, before any Search Console report; their cost was one hour of writing, once.

Prioritization

1. What stops indexing (blocks, noindex, 5xx, blocked bots): today.
2. What duplicates or dilutes (duplicates, canonicals, internal redirects): this week.
3. What slows things down (speed, resources, heavy pages): this month.
4. What improves discovery (sitemap, structure, orphans): this month.
5. What concerns hygiene (headers, structured data, hreflang): in the next two months.

Every fix is verified: URL inspection after correction, validation in the indexing report, comparison of crawl stats after two weeks.

Frequently asked questions

1. How long does a technical audit take? For a company website, one day; for a store with thousands of pages, two or three, plus the logs.
2. Can I do the audit myself? The basic checks, yes, with Search Console and a free crawler; interpretation and prioritization are the part that requires experience.
3. How often? Quarterly, plus after every launch; weekly automated checks cover the time between audits.

Maintenance

• Weekly automated check of the main pages (200 code, no noindex, correct canonical), from a curl script, with an email alert.
• The indexing report and crawl stats, monthly.
• Full audit quarterly and after any launch (theme, modules, migration).
• A site change log, so errors can be linked to their cause.

The full list of indexing errors, with the diagnosis and fix for each one, is in the article on common indexing errors; for WordPress, the specifics are in the guide to technical SEO for WordPress. The audit above, applied to your site and delivered as a priority list, is our technical SEO service.

Sources

• Search Console Help, page indexing report: statuses and reasons for non-indexing.
• Google, HTTP status codes and network errors: how Google treats each code and connection error.

Source references

• Ajutor Search Console, raportul de indexare a paginilor
• Google, codurile de stare HTTP și erorile de rețea