{"id":13483,"date":"2026-10-09T07:35:00","date_gmt":"2026-10-09T05:35:00","guid":{"rendered":"https:\/\/www.lukaswojcik.com\/blog\/?p=13483"},"modified":"2026-09-25T09:45:53","modified_gmt":"2026-09-25T07:45:53","slug":"canonical-hreflang-and-noindex-why-a-page-disappears-from-the-index","status":"publish","type":"post","link":"https:\/\/www.lukaswojcik.com\/blog\/en\/digital-marketing\/tutorials-en-digital-marketing\/canonical-hreflang-and-noindex-why-a-page-disappears-from-the-index\/","title":{"rendered":"Canonical, hreflang and noindex: Why a Page Disappears from the Index"},"content":{"rendered":"

A page can be perfect and invisible. Written, published, linked, fast – and absent from search results, because one line somewhere said it should be.<\/p>\n

Four signals decide this, and they are not read in isolation. They are read in a fixed order, each one only reached if the previous one allowed it, and that order produces at least one result that looks backwards.<\/p>\n

\n\"Four
Each step is only reached if the one before it allowed it – which is where the trap sits.<\/figcaption><\/figure>\n

What the four signals are<\/h2>\n

robots.txt<\/strong> is a file at the root of a site that says which addresses a crawler may fetch. It governs fetching, and nothing else.<\/p>\n

The robots meta instruction<\/strong> sits in the page itself, or in a header of the response, and says whether the page may be listed. noindex<\/code> means: read it, do not list it.<\/p>\n

The canonical<\/strong> names which address is the real one when several show the same content. A page pointing at another address hands its search ranking over to that address.<\/p>\n

hreflang<\/strong> tells a search engine which language versions of a page exist, so a Polish visitor gets the Polish page rather than the English one.<\/p>\n

The first two decide whether a page is listed. The other two decide which address gets listed and to whom it is shown.<\/p>\n

The canonical that points somewhere else<\/h2>\n

A canonical is a suggestion, and it is usually followed.<\/p>\n

Its purpose is duplicates. A shop article reachable under three addresses should be counted once, not three times, and the canonical names the one that counts. Every version points at the same address, and the ranking gathers there instead of splitting.<\/p>\n

The failure is small and quiet: a page pointing at a different address than its own. It happens when a template is copied and the canonical is left hardcoded, when a staging system is cloned, or when a translation plugin points every language at the English version. The result is a page that exists, works, and gives everything it earns to another address.<\/p>\n

A canonical also has to be an absolute address. A relative one is interpreted, and interpreted against the wrong base it points somewhere nobody intended. The check for this is worth doing precisely because it produces no visible symptom.<\/p>\n

When a block hides the noindex<\/h2>\n

This is the case that reads backwards, and it is the most common serious mistake of the four.<\/p>\n

A page is meant to disappear from search results. Two things are done at once: noindex<\/code> is put in the page, and the address is blocked in robots.txt. Both feel like the same intention, expressed twice.<\/p>\n

They are not. robots.txt prevents fetching. A crawler that may not fetch the page never reads its content – and noindex<\/code> is in the content. The instruction to remove the page cannot be received, because the way to receive it has been blocked.<\/p>\n

The page can then stay in the index, listed from links pointing at it, usually without a description because nothing could be read. It is a familiar sight in search results, and it is exactly this mistake.<\/p>\n

The correct order is the other way around: allow fetching, let the noindex<\/code> be read, wait until the page is gone – and only then, if desired, block the address.<\/p>\n

\n\"A
\n<\/figure>\n

hreflang and the reference back<\/h2>\n

hreflang is the most laborious of the four, because a single page can never be right on its own.<\/p>\n

The rule is mutual reference. If the German page names the English one, the English one has to name the German one back. A reference in one direction only is ignored – not partially honoured, ignored – and the language versions stay unconnected.<\/p>\n

Two further rules cost little and are skipped often. Every page in a group has to name itself as well as the others; a group without a self-reference is invalid. And one entry should carry x-default<\/code>, for visitors whose language is not among the versions – without it, a visitor from a fourth country lands wherever the search engine guesses.<\/p>\n

An honest limit belongs here: whether the other side really points back can only be established by fetching the other side too. A check of a single page can read what that page claims and can see whether the self-reference is there – it cannot confirm the return reference. Any tool reporting hreflang as “correct” from one page has not checked the part that fails most often.<\/p>\n

The group in robots.txt that overrules the rest<\/h2>\n

robots.txt looks like a list of rules and is really a set of separate rulebooks, one per crawler.<\/p>\n

A file may contain a group for *<\/code> and another for a named crawler. It is tempting to read them as layers: the general one applies, the specific one adds to it. That is not how it works. A crawler with a group of its own reads only that group and ignores the group for *<\/code> entirely.<\/p>\n

So a file blocking \/private\/<\/code> for everyone, and containing a separate group for one search engine that does not mention \/private\/<\/code>, allows that one search engine into \/private\/<\/code>. The general block is not weakened – it simply does not apply to a crawler that brought its own rules.<\/p>\n

This is worth stating plainly because the intuition points the wrong way, and because a rule that applies to almost everyone tends not to be tested against the exception.<\/p>\n

What one fetch shows<\/h2>\n

All four signals arrive with a single request, plus one for robots.txt. Reading them together is what makes the picture, because each one only matters in the light of the others.<\/p>\n

Four questions: does the canonical point at the page itself, or somewhere else. Is there a noindex<\/code>, in the page or in a header. Does robots.txt allow fetching at all – and by which group’s rules. And do the hreflang entries include a self-reference and an x-default<\/code>.<\/p>\n

The indexing inspector<\/a> answers all four for any address. It follows the redirects, reads the signals from headers and page alike, evaluates robots.txt with the correct group precedence, and marks separately what it cannot know from one page – the reference back above all.<\/p>\n

The useful result is one sentence: whether this address can be listed, and if not, which of the four signals prevents it.<\/p>\n

\n

Questions and answers<\/h2>\n

What happens when the language versions carry hreflang but their canonical points at the English page?<\/h3>\n

The two signals contradict each other. hreflang says: this page is the German version of a group. The canonical says: the real address is the English one, and this page is a duplicate. If the search engine follows the canonical, as it usually does, it lists the English address and hands it the German page’s ranking. A visitor from Germany then gets the English page, which is exactly what hreflang is meant to prevent.<\/p>\n

A translation is not a duplicate, even though it shows the same content in another language. The correct setup is therefore for every language version to carry a canonical pointing at itself, and for the hreflang entries to name exactly those canonical addresses. If an hreflang entry points at an address whose own canonical points elsewhere, the group refers to a page that hands its ranking over to another address.<\/p>\n

A translation plugin is often behind this, and then every language is affected at once. Checking each version shows it quickly, because the canonical then does not point at the page’s own address.<\/p>\n

How can a PDF be kept out of the index when no meta tag can be written into it?<\/h3>\n

Through a header on the response: X-Robots-Tag: noindex<\/code> works like the robots meta instruction and applies to any file type. The same order as for pages holds here: robots.txt must not block the file, otherwise the header is never read either.<\/p>\n<\/div>\n

\n

Sources<\/h2>\n