Search engine optimization, oriented

Crawling is the fetch: a program downloads pages it finds from links on pages it already knows and from sitemaps you supply. Rendering happens here too — the crawler runs the page's JavaScript in a recent build of Chrome. Indexing is the analysis, where closely resembling pages are grouped so one representative can be chosen. Serving is decided per query, not by a fixed score attached to your site.

Collapsing those into one funnel is the field's founding mistake. Suppose your server log shows the fetch, the page renders fine, and a week later it returns nothing in search. Nothing is broken and there is nothing to resubmit: Crawled - currently not indexed is a documented state meaning exactly that — fetched, and deliberately not indexed, with an explicit note that resubmitting achieves nothing. Discovered - currently not indexed is one step earlier: known, not yet fetched.

Search Essentials collects the minimum technical requirements, the spam policies and the recommended practices. It also carries the sentence most advice omits: satisfying every one of them still does not mean a page will be crawled, indexed or served. Nor is any of it purchasable — no payment buys more frequent crawling or a higher position.

Rule: name the stage before naming the fix, because "we don't rank" is at least three unrelated bugs wearing one complaint.

The assumption that a site gets submitted somewhere has no mechanism behind it. Discovery runs on two paths — links from pages already known, and sitemaps — and a page that neither reaches may never be found.

A sitemap is a discovery aid and nothing stronger: listing a page does not guarantee it will be crawled or indexed, and the documentation is frank that many sites need none. Under roughly 500 pages, thoroughly linked from the homepage and without heavy video or news content, walking your links finds everything.

Fetch volume is two things multiplied: a capacity limit — what your server absorbs without degrading — and demand, reflecting size, update frequency and quality. Fast consistent responses raise capacity; latency, server errors and rate-limit responses lower it. You can ask for less crawling; there is no documented way to ask for more.

Crawl budget is the rest of the folklore. The guide documenting it scopes itself out of most people's lives: it is for sites above a million pages changing weekly, or ten thousand changing daily. If your pages are crawled the day you publish them, it does not apply. And suppose you mark ten thousand filtered URLs noindex to protect that budget — each is still requested and only then dropped, so the fetch was spent anyway.

Rule: crawl rate is earned from response times and wasted URLs, never requested.

A robots.txt file governs crawler traffic, mainly so a site is not overwhelmed with requests. The documentation is explicit that it is the wrong tool for keeping a page out of results: when a disallowed URL is linked from elsewhere, the address and publicly available text such as anchor text can still be listed — indexed without its content.

Removal is a different tool. A noindex rule, as a robots meta tag or an X-Robots-Tag response header, drops the page once the crawler reads it. The header form is what works for a file with no head element, such as a PDF.

Now the trap. Suppose you want /private/ gone, so you belt-and-brace it:

Applied to the same page these cancel out. The crawler never fetches /private/, so it never sees the tag, so the URL keeps appearing — and the harder you block, the more permanent the listing becomes. Deleting the disallow line is the fix: for noindex to work the page must be reachable.

Two mechanics catch people out. The file applies only to its own host, protocol and port, so an apex domain and its www form need their own. And it is not access control — a disallow list is a public catalog of what you would rather nobody read.

Rule: robots.txt answers whether a crawler may fetch; noindex answers whether it may list. Never ask one to do the other's job.

The sequence is fetch, queue, render, index. Pages returning a success status enter a rendering queue that may hold them for seconds or considerably longer; a non-success status may skip rendering entirely. So "works in my browser" and "was indexed" are separated by a wait you do not control, in an environment that is not yours.

Blocked resources are the classic silent failure. Suppose robots.txt disallows /assets/: the crawler will not fetch your bundle and will not run the script inside it, so the page is indexed as whatever the initial markup contained — frequently an empty shell, with no error anywhere to tell you.

The second failure is assumed state. Local storage, session storage and cookies are all cleared between page loads, and permission requests are declined. Content that appears only for a returning visitor, or after a location prompt, does not exist for the crawler — and neither does content revealed only by a scroll or a click.

Two structural rules follow. Routes need real URLs through the History interface rather than fragments, which are generally not supported for changing content. And a client-rendered "not found" returning a success status reads as a soft error page, so route to a URL that genuinely returns 404 or inject noindex.

Finally, parity: indexing uses the mobile rendering, so anything absent from the small-screen version is content you may not be indexed for. Collapsing it into an accordion is fine; deleting it is not.

Rule: verify against the rendered output, never the source you wrote.

Duplicates rarely come from plagiarism. They come from parameters, session identifiers, the secure and insecure forms of a URL, the www and apex forms, and trailing slashes — one page reachable a dozen ways. The published position is unusually relaxed: inefficient, and not a manual action.

The signals are ranked: a permanent redirect is a strong signal that its target should be canonical, rel="canonical" is likewise strong, and inclusion in a sitemap is weak. Secure URLs are preferred over insecure equivalents, and linking internally to one form consistently helps.

Now suppose you declare the clean URL canonical while every internal link, your sitemap and a stray redirect point at the parameterized one. The declaration loses, and a report status names that outcome precisely: Duplicate, Google chose different canonical than user. Two nearby mistakes make it worse — robots.txt cannot canonicalise, because a blocked URL may still be listed without content, and noindex does not consolidate, it removes.

Because redirects carry the strongest signal, status codes matter. A 301 or 308 is permanent and reads as canonical intent; 302, 303 and 307 are temporary and do not. Server-side beats a meta refresh, which beats a script-driven hop; up to ten hops are followed, so long chains are a real risk.

On URLs the advice is smaller than the industry implies: readable words beat long numeric identifiers, and hyphens beat underscores. Keywords in a domain or path alone have hardly any effect beyond appearing in the breadcrumb.

Rule: point every signal at one URL, then let the cluster form.

The mistake is treating the title element as a slot to fill rather than a label that gets judged. A title link can also be drawn from the main visible heading, other heading elements, og:title, prominently styled text, anchor text in links pointing at the page, and site-level structured data — so a rewrite is a defect report. Suppose four hundred pages carry a title of site name plus category: it is boilerplate, it distinguishes nothing, and something more useful gets generated instead. The same happens to half-empty titles, stuffed variants of one phrase, and titles that do not describe the content.

Snippets work similarly, assembled from page content matching the query, so one page shows different snippets for different searches. A meta description is used only sometimes — when it describes the page more accurately than text taken from the page. Write it anyway; the documentation discusses it only as snippet material, and nosnippet, max-snippet and data-nosnippet are the controls if you need them.

Then the refusals, addressed head-on in the published starter guidance. The keywords meta element is not used. There is no magical word count, minimum or maximum, and no keyword density figure appears anywhere in the guidance — there is no published target to hit. Heading elements used out of order do not matter to search. Subdomain versus subdirectory is a business decision.

Rule: write the title a person needs to tell your pages apart, and stop optimizing what the documentation says is inert.

The mistake is assuming anything clickable is a link. The requirement is narrow:

That is followable — including when it also carries a click handler, and when script inserts it. These are not: an anchor with a framework routing attribute but no href, a span given an href, an anchor whose whole behavior is an onclick, and an href holding a script expression rather than an address.

Suppose your category filters are clickable div elements. Every filtered view is reachable by a person and by nothing else, so an entire section has no path into it — and no report says so, because from the crawler's side those pages were never mentioned.

Anchor text is the context for the destination, which is why "click here" and "read more" waste the strongest hint you can give. Keep it descriptive and short; stacked adjacent links dilute each other. Internal links are also the structural map: anything you care about should be linked from at least one other page of yours.

Images are read from alternative text combined with image recognition and the page's own content, including captions and nearby text. Filenames contribute a little, so a descriptive name beats a camera's serial output; keep the image at a stable URL so it can be cached and reused.

Rule: if it is not an anchor with an href it is not a path, and if the message is only in pixels it is not content.

Structured data is a standardized way to state what a page is about instead of leaving it to be inferred. JSON-LD in a script element is the recommended form because it is easiest to maintain at scale; microdata and RDFa are also read.

The vocabulary itself is a hierarchy of types, each carrying properties. It is maintained independently of any search engine and says nothing about which types an engine supports, so a type existing there is not evidence anything will render it.

Two asymmetries are the whole lesson. Omitting a required property disqualifies the result outright; including every required property makes the page eligible and stops there. And the markup must describe content the user can actually see. Suppose you add review stars to a product page showing no reviews: you have annotated information that is not on the page, which risks losing eligibility or drawing a manual action rather than winning stars.

Video shows why markup alone is never the story. To be indexed, the page holding it must itself be indexed, the file must be fetchable rather than blocked, and there must be a valid thumbnail at a stable URL. Only then does VideoObject markup influence how it appears and unlock features such as key moments. A video that loads only after a swipe or a tap fails before markup is consulted.

Rule: markup describes what is on the page; it never adds what is not, and it never promises a result.

The published list of ranking systems is long and deliberately plural — neural matching, passage ranking, freshness, deduplication, site diversity, link analysis, all operating at page level. Several famous ones no longer exist separately: the helpful content system was absorbed into core ranking in March 2024, as earlier quality and link systems were in 2015 and 2016. Chasing "the update" therefore chases something that has usually dissolved into the whole.

The volume belief is the costlier one. Suppose you split one strong answer into six near-identical posts to cover more phrasings: deduplication exists precisely to stop several of them being shown, so they compete with each other instead of adding up.

The four-letter quality shorthand is the field's most over-claimed term. It stands for experience, expertise, authoritativeness and trustworthiness, and the documentation's own answer to whether it is a ranking factor is no — it describes what a mixture of signals tries to identify, and what human evaluators assess when checking the systems work. What is documented instead is a self-assessment: who made this, how, and why. Is authorship evident, is any automation disclosed, and is the page primarily meant to help a person rather than win a position.

On links the policy is specific: buying or exchanging them, generating them automatically, requiring one in a contract, paid articles that pass credit, and keyword-stuffed footer or widget links. Paid links are permitted when qualified as sponsored or not followed.

Rule: prefer the mechanism the documentation describes over the acronym built on top of it.

The three are Largest Contentful Paint, good at 2.5 seconds or less; Interaction to Next Paint, good at 200 milliseconds or less; and Cumulative Layout Shift, good at 0.1 or less. All are assessed at the 75th percentile of loads, split across mobile and desktop, so an average hides exactly the visitors who are suffering.

Field and lab numbers answer different questions. A scripted run on chosen hardware is repeatable and useful while developing, and it cannot stand in for what real visitors experienced. One consequence catches everyone: a tool loading a page without a user cannot measure Interaction to Next Paint at all. That is an absent value, not a good one.

On proportion the documentation says the opposite of the folklore, three ways: there is no single signal here, strong scores in a report do not guarantee a top position, and the most relevant content is still shown even when its page experience is sub-par. Suppose your page clears all three thresholds while answering the question worse than the slower page below it — you have not bought the position, and further tuning will not buy it either.

The rest of the published self-check rounds it out: served securely, usable on a phone, without distracting advertising or intrusive interstitials, main content clearly separated.

Rule: fix page experience because a person feels it, never as a substitute for being the better answer.

Start with the metrics, because the common mistake is reading average position as a rank you hold. It takes your best position for each impression and averages those — described in the documentation itself as complex and capable of misleading, since position numbering varies by device and result type. An impression is a link to your site being shown; a click is a visit leaving the results for your page, and returning to click the same link again still counts once; click-through rate is clicks divided by impressions. Counting also depends on grouping, so one appearance can be a single impression per site and several per page — which is why filtered totals refuse to reconcile.

Diagnosis follows the split. Suppose impressions hold steady while clicks fall by a third. Your position did not move and your content is not the suspect; the result's appearance changed — your own headline and snippet, or a richer competing result absorbing the attention. If impressions and clicks fell together, the cause is upstream instead: indexing, serving, a technical break, a policy issue, or genuinely reduced demand.

Then scope it. A site-wide pattern points at the indexing report; a specific group of pages points at inspecting those URLs. Widen the window to sixteen months first, because seasonal demand looks identical to a penalty at three months. The advice against acting is most often skipped: a page already performing well should not be radically changed over small movement.

Rule: identify which number moved before deciding what to change.