How Google Search Works: Crawl, Index, Rank

Google Search works in three stages: crawling (Googlebot discovers URLs and downloads them), indexing (Google analyses the page, deals with duplicates and decides whether to store it) and serving (Google ranks indexed pages for each query). Every page has to pass the stages in order, and most "my page is not on Google" problems are a failure at one specific stage. If you can tell which stage a URL is stuck at, the fix is usually obvious.

This guide walks through each stage the way Google documents it, with the practical checks that tell you where a page sits.

The short version in one table

Stage	What Google does	What can stop a page here	How to check
Discovery	Learns that a URL exists, from links or sitemaps	No links to the page, not in a sitemap	URL Inspection says "URL is unknown to Google"
Crawling	Googlebot requests the URL	robots.txt block, server errors, slow server, low crawl priority	"Discovered - currently not indexed", Crawl stats report
Rendering	Runs JavaScript in headless Chrome	Blocked JS or CSS files, content that needs user interaction	URL Inspection, View crawled page, HTML tab
Indexing	Analyses content, picks a canonical, decides to store or skip	noindex, duplicate content, thin content, soft 404	"Crawled - currently not indexed", canonical fields in URL Inspection
Serving and ranking	Matches indexed pages to a query and orders them	Weak relevance, stronger competitors, poor quality signals	Performance report: impressions and average position

Google is explicit that none of this is guaranteed. Its own documentation says that even if a page follows every guideline, Google does not promise to crawl it, index it or serve it.

Stage 1: discovery and crawling

How Google finds URLs

There is no central registry of web pages. Google has to find each URL, and it has two main ways of doing that:

1. Links. Googlebot fetches a page it already knows, extracts the links and adds new URLs to its list. This is the main discovery method on the web.
2. Sitemaps. You give Google a list of URLs, through Search Console or a Sitemap: line in robots.txt.

A URL that has no links pointing at it and appears in no sitemap can stay invisible to Google forever. That is the orphan page problem, and it is one of the most common reasons for "URL is unknown to Google".

Links need to be real links to count. Google's documentation says it can reliably follow only <a> elements with an href attribute:

<!-- Crawlable -->
<a href="/pricing">Pricing</a>

<!-- Not reliably crawlable -->
<span onclick="location.href='/pricing'">Pricing</span>
<a onclick="goTo('pricing')">Pricing</a>

What Googlebot is

Googlebot is the name of Google's crawler. It runs from many machines, and there are two main variants: Googlebot Smartphone and Googlebot Desktop. With mobile-first indexing, the large majority of crawling is done as the smartphone crawler, and the mobile version of your page is the one Google indexes. If your mobile layout hides content or links that the desktop layout shows, Google mostly sees the mobile version.

Googlebot is polite by design. It adjusts how fast it fetches based on how your server responds. Quick, healthy responses let it fetch more. Slow responses and 5xx errors make it back off. This is the "crawl capacity" half of crawl budget. The other half is demand: how much Google wants your URLs, which depends on popularity, how often the content changes and how good Google thinks the site is.

What blocks crawling

• robots.txt. A Disallow rule that matches the URL stops Googlebot from fetching it. Test any URL with the Robots.txt Tester.
• Server errors. 5xx responses and timeouts. If robots.txt itself returns a 5xx, Google treats the whole site as disallowed for a while.
• Login walls. Googlebot does not log in. Anything behind authentication returns 401 or 403 and is not crawled.
• Low priority. Google knows the URL and has decided not to spend a fetch on it yet. In Search Console this is "Discovered - currently not indexed", and on big or new sites it is the most common state. See Discovered, currently not indexed.

One detail that trips people up: robots.txt controls crawling, not indexing. A blocked URL can still show up in results (without a description) if other pages link to it. If you want a page kept out of the index, let Google crawl it and serve a noindex. The differences are covered in noindex vs robots.txt vs canonical.

Rendering

After the fetch, Google renders the page with a recent version of Chrome, executing JavaScript much like a browser would. Rendering matters because many sites produce their content, links or even their meta tags with JavaScript.

Two practical consequences:

• Rendering is queued. Google's documentation notes that a page may wait before it is rendered, so links and content that exist only after JavaScript runs are found later than those in the initial HTML.
• If robots.txt blocks the JavaScript or CSS files the page needs, Google renders a broken version and indexes that.

If your site depends on client-side rendering, read JavaScript SEO: rendering and indexing. The quick test is URL Inspection > Test live URL > View tested page > HTML. If your main content is not in that HTML, Google does not have it.

Stage 2: indexing

Crawling gets the bytes. Indexing is where Google works out what the page is and whether to keep it. Google's documentation describes this stage as processing and analysing the text, key tags and attributes such as the <title> element and alt attributes, images, videos and more.

What happens during indexing

Parsing. Google reads the content, headings, title, structured data, links and meta tags, including <meta name="robots"> and the X-Robots-Tag header. A noindex found here ends the process: the page is dropped.

Duplicate detection and canonicalisation. Google groups pages with the same or very similar content into a cluster and picks one as the canonical, the version that may be shown in results. Your rel="canonical" tag, redirects, sitemap entries and internal links are all signals in that decision, and Google can overrule the tag when other signals disagree. The rest of the cluster becomes alternates, reported in Search Console as "Alternate page with proper canonical tag" or "Duplicate, Google chose different canonical than user". The full mechanics are in the canonical tags guide.

Signal collection. Google stores information about the canonical page and its cluster: language, the country the content is local to, usability and so on. These are used later during serving.

The keep or skip decision. This is the part people find frustrating. Google says indexing depends on the content of the page and its metadata, and lists common problems: low quality content, robots meta rules that disallow indexing, and site design that makes indexing difficult. In practice, a page that is technically fine can still be skipped because Google does not think adding it to the index is worth it. That is what "Crawled - currently not indexed" means, and the fix is nearly always about the page and the site around it, not about resubmitting. See Crawled, currently not indexed.

What the index actually is

The Google index is a very large database spread across many machines. Think of it like the index at the back of a book: for each term, a list of documents in which it appears, plus a lot of extra information about each document. A page that is not in this database cannot appear for any query, no matter how good it is. That is why indexing problems are worth fixing before anything else in SEO.

How to check indexing properly

• URL Inspection in Search Console is the authoritative answer for a single URL. "URL is on Google" means indexed. The URL Inspection tool guide explains every field.
• The Page indexing report gives the totals and the reasons for non-indexed URLs across the property.
• The site: operator is a rough check only. Google says site: results are not a complete list of indexed pages, so a missing result there is a hint, not proof.

For a quick pre-check of whether a URL is even allowed into the index (status code, robots.txt, noindex, canonical), run it through the Indexability Checker.

Stage 3: serving and ranking

When someone searches, Google looks up matching pages in the index and orders them with its ranking systems. Google says relevance is determined by hundreds of factors, and that results depend on things like the user's location, language and device. A search for "bicycle repair shops" returns different results in Paris and in Hong Kong.

Google does not publish a formula, but its documentation on how Search works groups the main considerations like this:

Consideration	What it means in practice
Meaning of the query	Google works out intent: is this person looking for a definition, a product, a place, a fresh news story?
Relevance of content	Does the page actually cover what the query asks? Matching words is the basic signal, but Google also looks at whether the page covers the topic.
Quality of content	Signals that help identify expertise, authoritativeness and trustworthiness. Links from other prominent sites on the topic are one of them.
Usability	When other things are roughly equal, pages that are easier to use do better. Page experience, including Core Web Vitals, sits here.
Context and settings	Location, search history and search settings.

Two points are worth stating clearly because they are misunderstood so often.

Indexed does not mean ranking. An indexed page is eligible to appear. Whether it appears on page one or page nine depends on how it compares with everything else that is eligible. If your page is indexed and gets no impressions, you have a ranking problem, covered in indexed but not ranking.

Search features depend on the index too. Rich results, sitelinks, image results and video results are all built from what Google stored at indexing time. Structured data that Google could not parse, or images blocked from crawling, never make it into those features.

Where things go wrong: a diagnostic order

When a page is not showing up, work through the stages in order. Skipping ahead wastes time, because a later stage cannot succeed if an earlier one failed.

1. Is the URL known? Inspect it. "URL is unknown to Google" means discovery failed. Add internal links and put it in the sitemap. See URL is unknown to Google.
2. Can it be crawled? Check robots.txt, the HTTP status (it should return 200) and that the server responds quickly. The Crawl stats report under Settings shows host problems.
3. Does the rendered HTML contain the content? Live test, then View tested page.
4. Is it allowed in the index? Look for noindex in the HTML and in the X-Robots-Tag header. Check that the canonical points to the page itself.
5. Did Google choose it? If the state is "Crawled - currently not indexed" or a duplicate state, the question becomes quality and uniqueness.
6. Is it indexed and invisible? Then look at the Performance report for impressions and average position, and work on relevance, content and links.

How long each stage takes

Google gives no fixed timings, and anyone who quotes a guaranteed number is guessing. What Google does say: crawling can take anywhere from a few days to a few weeks, and requesting indexing does not make it instant. Established sites with frequent updates are crawled within hours or days. New sites with no links can wait weeks. There is more detail in how long Google takes to index.

What you control:

• Discovery speed. A sitemap with accurate lastmod values, plus internal links from pages Google already crawls often.
• Crawl speed. A fast, stable server and no wasted crawling on parameter junk.
• Notification. The Request indexing button for a handful of URLs, the Indexing API for the page types Google supports (JobPosting and BroadcastEvent), and IndexNow for Bing and other participating engines. Google does not use IndexNow.

TurboIndex automates that last part: it reads your sitemaps, submits URLs within Google's daily quotas and then uses the URL Inspection API to report which stage each URL has reached. It cannot make Google index a page, and nothing can, but it removes the manual checking.

Common misconceptions

"Submitting a sitemap gets my pages indexed." A sitemap is a discovery hint. It gets URLs onto Google's list. It does not influence the keep or skip decision.

"Google did not index my page, so there must be a technical error." Often there is none. Google chooses not to index a large share of what it crawls, and the Page indexing report calls that out as normal. The aim is to have your important canonical pages indexed, not 100 percent of URLs.

"Blocking a page in robots.txt removes it from Google." It stops crawling. If the page was already indexed, it can stay indexed, and Google can no longer see a noindex tag you add to it.

"More crawling means better rankings." Crawl frequency is not a ranking signal. Google says so directly in its crawl budget documentation. Crawling is a prerequisite, not a reward.

"Google Ads customers get crawled faster." Google states that it does not accept payment to crawl a site more often or rank it higher.

What to do with this

Keep the three stages in your head as separate problems with separate tools. Discovery and crawling are fixed with links, sitemaps, robots.txt and server health. Indexing is fixed with clean canonical signals, no stray noindex and pages that deserve a place in the index. Ranking is fixed with relevance, quality and reputation. If you are setting up a site from scratch, the new website SEO checklist puts these steps in order for the first 30 days.