How to Get Google to Crawl Your Site Sooner

Google crawls your site when it has a reason to: a link from a page it already visits, a sitemap entry with a fresh <lastmod>, a request through Search Console, or an Indexing API notification for a job or livestream page. Nothing forces a crawl. What you can do is give Googlebot more reasons to come, remove the things that slow it down, and check when it actually came so you stop guessing.

Crawling is not indexing

Crawling means Googlebot fetched the page. Indexing means Google decided to keep it and can show it in results. The two are separate steps with separate reasons for failure, and Search Console names them separately: "Discovered - currently not indexed" is a crawl problem (Google knows the URL, has not fetched it), while "Crawled - currently not indexed" is an indexing problem (Google fetched it and passed). This article is about the first step. If your pages are being crawled and still not indexed, the content is the issue, and more crawling will not change the answer.

What makes Googlebot come

Links from pages Google already crawls

Following links from known pages is Google's main discovery method and the strongest crawl signal you control. A new page linked from your homepage is usually fetched the next time Google crawls the homepage, which on an active site is daily. The same page linked only from page nine of an archive can wait for weeks.

So when a URL is not being crawled, start by asking where it is linked from. Links > Internal links in Search Console lists inbound internal links per URL. Zero or one is the usual answer for a page that Google ignores. Links from other sites work the same way with more weight, because they also raise how often Google wants to visit your host at all.

A sitemap with truthful lastmod dates

An XML sitemap submitted under Indexing > Sitemaps gives Google a list of URLs and a date per URL. Google uses <lastmod> to decide what to recrawl, but only on sites where the date is consistently accurate. Sites that stamp today's date on every URL on every request teach Google to ignore the field, and lose the one mechanism that gets changed pages recrawled without a request.

The rules: one date per URL, changed only when the page content changes, 50,000 URLs and 50 MB per file, a sitemap index for more. <priority> and <changefreq> are ignored. Run the file through the sitemap validator before you rely on it. And note that the sitemap ping endpoint was switched off at the end of 2023, so resubmitting the same file does nothing; changing lastmod on the changed URLs is what counts.

Request indexing

For one URL, the URL Inspection tool's Request indexing button puts the page in a priority crawl queue. The allowance is small (Google does not publish it; around ten per property per day is what people see) and Google says repeated requests for the same URL do not speed anything up. Use it for the pages that matter this week.

The Indexing API

Google's Indexing API accepts URL_UPDATED and URL_DELETED notifications, 200 per day per Cloud project, and schedules a crawl for each. Google's documentation restricts it to pages with JobPosting or BroadcastEvent structured data and names quota evasion with several accounts as abuse. For a job board it is the right tool; for other sites it is unsupported.

What does not reach Google

• IndexNow. Bing, Yandex, Naver, Seznam and Yep listen. Google does not.
• Pinging. The sitemap ping is gone, and Google never accepted pings for arbitrary URLs.
• Resubmitting an unchanged sitemap. Google re-reads it on its own schedule either way.
• Social posts. They may get a page discovered, but a link from a crawled page is the thing doing the work, not the share.

What keeps Googlebot away

Giving Google reasons to come is half the job. The other half is not wasting the visits it makes.

Slow responses. Google adjusts its crawl rate to what your server can handle. If pages take seconds to answer, Googlebot fetches fewer of them per visit. Average response time is on the chart in Settings > Crawl stats.

Errors. A 5xx or a 429 makes Googlebot back off and retry later, and a run of them lowers the crawl rate for the whole host. Check the Host status section of Crawl stats, and read server errors and Googlebot if it is red.

Redirect chains and dead links. Googlebot follows up to 10 redirect hops, but every hop is a fetch spent on nothing. The same goes for 404s in your sitemap and internal links. The HTTP status checker tests a list of URLs and shows the final status and redirect count for each. TurboIndex applies the same idea before it asks Google for anything: its crawler, TurboIndexBot, fetches every new sitemap URL first, and a URL that redirects, answers 404 or carries noindex is never submitted (how it identifies itself is on the bot page).

robots.txt mistakes. A Disallow rule stops the fetch entirely. Blocked pages can still be indexed from links, without content, but they are never crawled. Test rules with the robots.txt tester before and after every deploy that touches the file.

Junk URLs. Faceted navigation, session parameters, calendar pages and internal search results can generate millions of URLs that are all technically crawlable. Google spends its visits on them and reaches your real pages less often. Block the patterns in robots.txt or stop linking to them.

Crawl budget: a big-site problem

Crawl budget is the number of URLs Googlebot can and wants to fetch on your site. Google's own guidance says it is a concern for large sites, around a million unique pages with content that changes weekly, or medium sites of more than ten thousand pages whose content changes daily. Below that, Google can crawl everything you have, and a page that is not being crawled has a different problem: nobody links to it, or the host is unhealthy, or the site has trained Google to expect little from new URLs.

If you do run a large site, crawl budget explained covers the levers: fewer junk URLs, faster responses, consistent lastmod, and consolidating duplicates.

How to check when Google last crawled

Guessing is the usual mistake here. Three places give you the date.

URL Inspection. Paste the URL into Search Console. The Last crawl field in the Page indexing section shows date and time; it is empty for a URL Google has never fetched. Crawled as shows the smartphone Googlebot for all sites now.

The URL Inspection API. The same data per URL, as lastCrawlTime, at up to 2,000 inspections per day per property. This is how you check a whole sitemap rather than one page. The field is absent for never-crawled URLs.

Server logs. The only source that shows every fetch, including the ones Search Console summarises away. Filter for the Googlebot user agent, then verify the IP address against Google's published ranges, because anyone can send that user agent string. Log file analysis for Googlebot shows how to do it and what to look for.

The Crawl stats report under Settings is the overview: total crawl requests per day, bytes downloaded, average response time, response codes, file types and crawl purpose, for the last 90 days, per host. It does not show individual URLs but it tells you at once whether Google's interest in the site went up or down.

What the date tells you is in when did Google last crawl my page. The short version: a page crawled yesterday and still not indexed has a content problem; a page not crawled for months is one Google has decided it does not need to revisit; a page never crawled has a discovery problem.

What "force" really means

People search for ways to force a crawl, and every method in this article is a request. Here is the honest ranking of what each one does:

Method	What it does	Limit
Internal link from a frequently crawled page	Discovery on Google's next visit to that page	None
External link from a crawled site	Discovery plus more crawl demand for your host	None
Sitemap lastmod	Flags a URL for recrawl, if your dates are trustworthy	50,000 URLs per file
Request indexing	Priority queue for one URL	Small daily allowance
Indexing API	Scheduled crawl	200 per day, job and livestream pages only
Fast, error-free server	Lets Google crawl more per visit	None

Nothing on the list makes Googlebot appear within the hour. Combined, they make a healthy site one where new pages are fetched within a day or two without anyone pressing a button.

What to do next

1. Inspect three URLs you think are not being crawled and read Last crawl. Confirm the problem exists.
2. Open Settings > Crawl stats and check host status, response time and the error share.
3. Link the uncrawled pages from your homepage or a section page.
4. Fix the sitemap: validate it, remove 404s and redirects, make lastmod honest.
5. Request indexing for the few pages that matter most, once each.
6. Come back in a week and compare the crawl dates.