Part of the Indexing Problems topic guide. Use the guide to move from this specific question to the full diagnostic workflow.
Server and access errors prevent Google from retrieving a reliable page response. The evidence may be an HTTP 5xx or 429 response, a 401/403 access restriction, a timeout, connection reset, TLS problem, DNS failure or an unavailable robots.txt file. Do not translate any one failed check directly into “not indexed”: first record the availability event, determine whether it is URL-level or host-wide, fix the responsible layer, and then observe a later crawl and index state separately.
For backlink providers and sellers, this distinction is essential. A third-party source page that cannot be reached at check time is unavailable; that is not the same as a verified missing link, a Search Console server-error status or a confirmed not-indexed result.
Server and access failures form the availability branch of the IndexVero indexing-problem diagnostic library. Start with the page-not-indexed diagnostic workflow when availability is only one possible branch. For an owned property, follow the URL Inspection evidence workflow to compare the recorded crawl with the current live response.
Error classes and what they establish
| Class | Example | What it establishes | What it does not establish |
|---|---|---|---|
| Server failure | 500, 502, 503, 504 | The server or an upstream component failed to provide a successful response | The permanent index state or the exact root cause |
| Rate limiting | 429 | The responding layer refused the request because of request-rate policy or load | That Googlebot is blocked intentionally or permanently |
| Authentication/authorization | 401, 403 | The request did not receive public access to the resource | Whether the restriction came from the origin, CDN, WAF, login or another rule |
| Network transport | Timeout, reset, incomplete response, TLS failure | No reliable page response completed | An HTTP status or page-content diagnosis |
| DNS | NXDOMAIN, lookup timeout, wrong routing | The hostname could not be resolved reliably at that time | A URL-level content or canonical issue |
| robots.txt availability | robots.txt returns 429/5xx or cannot be fetched | Google may slow or pause crawling while it cannot obtain an acceptable robots response | That a normal Disallow rule caused the failure |
Google’s network and DNS error documentation says that timeouts, connection resets and DNS errors are treated similarly to 5xx server failures for crawling. The Crawl Stats documentation separates robots.txt availability, DNS resolution and server connectivity in Host status.
Google documentation last checked: August 2, 2026.
If the server completes a nominally successful response but the rendered document is empty or error-like, continue with the Soft 404 guide; that is a content-response mismatch rather than a completed 5xx, DNS or transport failure.
Use an evidence ladder, not a single screenshot
- External availability check: exact URL, timestamp, response or transport failure, redirect chain and final URL.
- URL Inspection: recorded fetch status and crawl date for a URL in an authorized property; live test for the current response.
- Page Indexing report: affected status group and examples, noting that examples are not a complete URL inventory.
- Crawl Stats Host status: host-wide robots, DNS and connectivity trends.
- CDN, WAF, load-balancer and origin logs: the request path, response layer, latency, rejection rule and upstream error.
A live test that succeeds after a recorded error may show that the current response is healthy. It does not erase the older failure or prove that the problem was never intermittent.
URL-level or host-wide?
| Pattern | Likely scope to investigate | First evidence |
|---|---|---|
| One template or route returns 5xx | Application, database query, plugin or route-specific access rule | Origin logs and affected URL samples |
| Many hosts/routes fail together | DNS, CDN, WAF, load balancer, infrastructure or deployment event | Host status, infrastructure monitoring and edge logs |
| Googlebot receives 403 but normal users receive 200 | Bot protection, firewall or verification rule | WAF decision log and verified crawler identity |
| 429 increases with crawl volume | Capacity or rate-limit policy | Rate-limit counters, response headers and crawl-response trends |
| robots.txt alone is unavailable | robots route, CDN cache, security rule or origin handling | robots response history and Host status |
A safe triage decision tree
1. Confirm DNS and transport
Check that the hostname resolves to the intended infrastructure from more than one resolver and that TLS, routing and connection establishment complete. A DNS or transport failure occurs before page-content analysis.
2. Check robots.txt availability
A site does not need a robots.txt file, but the robots URL must return an acceptable response. Google documents 200, 403, 404 and 410 as successful robots.txt availability responses; 429 and 5xx are treated as connection problems. Do not confuse an unavailable robots file with a valid Disallow rule; use the robots and noindex guide to diagnose crawl instructions separately.
3. Locate the rejecting layer
Determine whether the response came from the CDN, WAF, reverse proxy, load balancer, application or upstream service. Preserve a request ID or timestamp so the same event can be found in logs.
4. Separate capacity from policy
A 429 can be a deliberate rate-limit response; a 503 can represent overload or maintenance. A 401/403 can be an intentional private-resource policy or an accidental block. Fix the layer that owns the behavior rather than merely rewriting the public error page.
5. Verify the public response
If the URL is intended for indexing, Google must be able to access a stable public representation without login. Test the current URL, then compare it with the recorded crawl evidence and the host-wide trend.
Provider and seller delivery states
| Observed state | Delivery report | Denominator treatment |
|---|---|---|
| Stable successful response, expected link present | Delivery verified at the check time | Eligible for the separate index-observation workflow |
| Terminal 4xx/5xx or access denied | Source unavailable or inaccessible, with exact response and time | Keep outside verified link-present delivery until resolved |
| Timeout, DNS or connection failure | Unavailable; no content or link conclusion produced | Exclude from binary indexed/not-observed claims until a valid check completes |
| Intermittent success and failure | Unstable; report both observations and the window | Do not select only the favorable sample |
| Third-party page inaccessible to the provider | Public access could not be verified | Do not assign a private Search Console reason |
The IndexVero measurement methodology keeps unavailable checks separate from exact, alternate/canonical and not-observed outcomes. This prevents infrastructure failure from being misreported as a Google indexing verdict.
What to fix by error family
- 5xx: correct application exceptions, upstream failures, deployment errors and exhausted capacity; return a stable success response only when the page is truly available.
- 429: review rate-limit thresholds, concurrency and serving capacity. Do not return a fake 200 error page to hide throttling.
- 401/403: decide whether the page is intentionally private. For a public indexable page, remove accidental authentication, CDN or WAF restrictions.
- Timeout/reset: trace latency and incomplete responses through the edge and origin; reduce expensive page generation and unstable dependencies.
- DNS/TLS: correct records, nameservers, routing and certificate/handshake problems at the responsible infrastructure layer.
- robots.txt unavailable: ensure the robots URL returns an acceptable response consistently and is not intercepted by a failing application route.
Recheck and close the incident
- Record the original failure class, URL, timestamp and evidence source.
- Confirm a stable response from the public edge and the origin where possible.
- Run URL Inspection live test for an authorized property.
- Review Crawl Stats for a host-wide trend rather than one successful request.
- Record the later crawl/index observation separately; do not claim that the fix caused indexing without stronger evidence.
The availability knowledge graph
hostname → DNS resolution → network/TLS connection → edge policy → origin response → rendered document → expected backlink → recorded crawl → index observation
A failure upstream prevents reliable conclusions downstream. If DNS fails, there is no page response to classify. If the document is unavailable, link presence cannot be verified. If a later check succeeds, it is a new observation—not a rewrite of the earlier incident.
For provider batches, connect these access states to the backlink delivery and indexing hub; an unavailable fetch remains outside verified delivery and observation denominators.
FAQ
Does one 500 error remove a page from Google?
One failed request does not establish a permanent result. Record whether the error was transient or repeated, review the property evidence and confirm a later stable response.
Is 429 the same as robots.txt blocking?
No. A 429 is a rate-limit response. A robots.txt rule is a crawl instruction. An unavailable robots.txt file returning 429 is a host-availability problem, not a valid Disallow rule.
Can a 403 page be indexed?
A 403 shows that the tested request was denied. The current or future index state depends on other evidence and Google’s ability to access a representation; do not use the 403 alone as the index verdict.
How should unavailable backlink checks be reported?
Report them as unavailable with the error class and timestamp. Do not convert them into “not indexed,” “link removed” or a private Search Console reason.
If the URLs you need indexed are backlinks on sites you don’t control, Search Console can’t submit them for you. Our backlink indexing service handles third-party URLs in bulk and verifies the outcome.
Leave a Reply