Why a Website Is Missing from Google: Checking Canonicals, Language Links and noindex

A published page does not automatically appear in search results. Crawl access, indexing instructions, duplicate content and language targeting serve different purposes. The following examples concern Google Search; other search engines may interpret signals differently.

robots.txt controls crawling, noindex excludes content from search results, canonical indicates the preferred URL for similar content, and hreflang identifies language or regional alternatives. These are not a rigid four-stage processing pipeline. Missing hreflang, for example, does not generally prevent indexing.

Canonical is a hint, not a redirect

Several URLs can serve identical or very similar content, such as a product with and without tracking parameters. A rel="canonical" pointing to the preferred product address can be correct. A canonical URL different from the current address is therefore not automatically an error.

Google treats the annotation as a strong signal alongside other evidence. Consolidation depends on the canonical URL actually selected. A canonical does not guarantee transfer of every ranking signal and does not redirect the browser.

Unintended targets are problematic, such as a forgotten staging address or a completely different page. Canonical annotations, internal links, redirects and sitemaps should consistently support the intended URL. Google supports relative canonical addresses such as /product but recommends complete URLs such as https://example.com/product. A complete URL avoids ambiguity about which host or base path supplies the missing parts.

noindex must remain accessible

A crawlable HTML page can contain <meta name="robots" content="noindex"> in the head. For PDFs and other resources, the response header X-Robots-Tag: noindex is suitable. A robots.txt block prevents Googlebot from reading a newly added instruction there.

A blocked URL may still appear in search results based on external links. For ongoing exclusion through noindex, the instruction should remain accessible. Blocking crawling after successful removal is therefore not a reliable continuation of this method.

Changes take effect after reprocessing, without a guaranteed date. Confidential content requires access control. Neither robots.txt nor noindex replaces authentication or permissions.

hreflang describes alternatives, not access restrictions

hreflang helps select suitable language and regional versions. A language code such as pl means Polish, not every visit originating in Poland. The annotation does not force a redirect or a particular search result.

With HTML link annotations, each version should list itself and the related alternatives using fully qualified URLs. Return links are necessary: a German-to-English link needs an English-to-German counterpart. A broken connection does not necessarily invalidate every other valid connection in the group.

x-default is a recommended fallback for unmatched language and regional settings, such as a language selector. Its absence alone does not invalidate a group. hreflang can also be supplied through HTTP Link headers or a sitemap; an HTML-only check does not fully discover those annotations.

Translations and canonical need joint checks

Fully translated main content is not duplicate content merely because the topic is the same. Separate German, English and Polish versions should each have an appropriate canonical URL in the same language. For several same-language duplicates, that URL may differ from the current address.

hreflang references should point to the intended canonical versions. Canonicalising every translation to English creates conflicting signals and can undermine the desired language selection. Automatic transfer of all rankings cannot be inferred.

robots.txt: Applicable groups and rules

Google uses the most specific matching User-Agent group. Equally specific groups are combined. The general * group does not automatically supplement a matching specific group.

For example, User-agent: * blocks /private/. A separate Googlebot group contains only a rule for /test/. Googlebot does not inherit that general /private/ block. Within the selected rules, matching paths and specificity matter; Allow wins an equally specific conflict.

A robots.txt file applies to its host, protocol and port. The main domain’s file does not automatically govern every subdomain. A disallow rule does not protect confidential files against direct access.

What the inspector can establish

The indexing inspector provides an initial snapshot. However, the current implementation does not cover all valid forms: canonical Link headers, some crawler-specific robots instructions and correct merging of repeated robots.txt groups are among the gaps. A green result therefore does not prove complete indexability.

Diagnosis also needs status codes, redirect chains, rendered pages, alternate language versions and any sitemap annotations. URL Inspection in Search Console shows Google’s stored state and selected canonical URL. A live test does not replace that indexed state, and even a technically accessible page has no indexing guarantee.