Programmatic SEO: Getting Pages Indexed at Scale

Getting programmatic pages indexed at scale depends far more on the pages than on submission methods. Google crawls a sample of a new template, decides whether pages like that deserve index space, and sets crawl demand for the rest accordingly. If the sample looks like the same page with a city name swapped, most of the set stays in "Discovered - currently not indexed" or "Crawled - currently not indexed", whatever you submit. If every page answers a real query with data of its own, the remaining job is engineering: links, sitemaps, server capacity and monitoring.

Know the policy line before building

Google's spam policies define scaled content abuse as generating many pages primarily to manipulate search rankings and not to help users, and state that it applies whether the content is produced by automation, by people or by a mix. Doorway abuse covers sets of pages made to rank for similar queries that all funnel users to the same destination, with city and region pages given as a typical example.

Programmatic sites that work well share these traits:

• Unique data per page that does not exist in the same form on other pages of the set: prices, specifications, statistics, availability, reviews, comparisons, results computed from a real dataset.
• Real search demand per page, even if small.
• A page that finishes the task. The visitor gets the answer there, without being pushed elsewhere.

A quick self-test: strip out the template and look at what is left on three random pages. One sentence with different nouns in it is the doorway pattern. A different table of real numbers on each one is fine.

Set indexability thresholds in code

Not every row in your database deserves an indexable page. Decide the rule in the template:

if listings_count >= 3 and has_unique_description:
    robots = "index, follow";  include_in_sitemap = true
else:
    robots = "noindex, follow"; include_in_sitemap = false

Typical thresholds: a minimum number of listings or data points, a minimum of unique text, the presence of key fields. Combinations that produce zero results should return a 404, which is what Google's documentation recommends for empty filter combinations. They should not return a 200 page saying "no results", since those become soft 404s.

This one step prevents the most common failure: 80 percent of the set is empty shells, and Google judges the template by them. See thin content and indexing.

Roll out in stages

Publishing 200,000 URLs on day one of a new domain is the fastest way to find out that Google will not crawl them.

Stage	Volume	What you are testing	Move on when
1	A few hundred of your best pages	Does Google index this template at all?	Most are indexed and some get impressions
2	A few thousand	Do internal linking and sitemaps scale?	The indexed share holds, crawl stats are healthy
3	Tens of thousands and up, in steps	Server capacity and crawl demand	Indexing keeps pace with publishing

If stage 1 stalls at "Crawled - currently not indexed", stop and fix the page before adding volume. More URLs of a rejected template lower the quality signals of the site as a whole.

Build the link architecture first

Sitemaps get URLs discovered. Internal links get them crawled and valued. Every programmatic page should be reachable through plain <a href> links within a few clicks of the homepage.

Hub pages. Build a hierarchy that mirrors the data: country > region > city, or category > subcategory > item. Each hub lists its children with crawlable links. Hubs should be useful pages in themselves, with summaries and aggregate data, not bare link lists.

Pagination that Google can walk. Use real URLs such as ?page=2 with <a> links, not buttons or infinite scroll alone. Link to more than "next": showing pages 1, 2, 3, 10, 20 and last cuts the depth of a long list drastically.

Cross-links. "Nearby cities", "similar products", "alternatives to X". These give every page several inbound links from related pages and spread crawl paths so that they do not all run through the hubs.

Links from strong pages. Link the top-level hubs from the homepage and the main navigation. If the site has editorial content with external links, link from there into the relevant hubs.

After launch, check that no page exists only in the sitemap. The comparison method is in orphan pages, and the principles in internal linking for indexing.

Sitemaps for large sets

The protocol limits are 50,000 URLs and 50 MB uncompressed per sitemap file. Use a sitemap index to tie files together. Beyond that:

• Segment by template and section, not at random. sitemap-cities-a.xml, sitemap-products-shoes-1.xml. In Search Console you can filter the Page indexing report by sitemap, which gives you the indexed share per template. That is the most useful metric you will have.
• List only indexable canonical URLs that return 200. Apply the same threshold logic as the robots tag.
• Use an accurate lastmod. Google says it uses lastmod when it is consistently and verifiably accurate. Set it when the data on the page really changes, not on every build. If every URL always shows today's date, the signal is ignored. See lastmod best practices.
• Add a separate sitemap for new pages holding the most recent additions, so fresh URLs are not buried among millions of old ones.
• Reference the sitemap index in robots.txt and submit it in Search Console.

Validate the output with the XML Sitemap Validator before submitting. A generated sitemap with a broken entity or a wrong hostname can sit unnoticed for months. Structure details are in sitemap index files for large sites.

Make crawling cheap

At this scale, crawl budget is real. Google's documentation describes it as the combination of a crawl capacity limit (how much your server can take) and crawl demand (how much Google wants your URLs). You influence both.

Capacity

• Serve pages quickly. Cache rendered HTML. A page that needs three seconds of database work on every request caps how fast Googlebot will go.
• Avoid 5xx and 429 responses under load. Google slows down when it sees them.
• Support If-Modified-Since or ETag and return 304 Not Modified for unchanged pages.
• Render on the server. Client-rendered pages need a second, queued rendering pass and additional requests for each page.

Demand, and waste

• Keep parameter variants, sorts and filters away from crawlers. See faceted navigation and indexing.
• One URL per page: consistent trailing slash, lowercase, with a self-referencing canonical.
• Return 404 or 410 for removed pages. Do not leave them at 200 with an empty template.
• Avoid redirect chains in internal links.

Monitor this in Search Console > Settings > Crawl stats: total requests per day, average response time, the split by response code, and the split by purpose (discovery or refresh). On a growing programmatic site you want discovery crawling to keep rising with a flat response time. Full background in crawl budget explained.

Differentiate the pages

Things that make templated pages distinct for Google and useful for people:

• Computed insights. "Median price here is 14 percent above the regional average" is unique to that page, and cheap to generate from data you already hold.
• Several data modules per page, so that pages differ by more than one field: a table, a chart, a related list, recent changes.
• Distinct titles and headings built from meaningful attributes. Not Best {keyword} in {city} repeated 40,000 times. See title tags and meta descriptions.
• User content, where it fits: reviews, questions, photos.
• Editorial content on the head pages. The top few hundred pages by demand can justify hand-written sections.
• Structured data that matches the entity: Product, LocalBusiness, JobPosting, Event, Dataset.

Near-duplicates show up in Search Console as "Duplicate, Google chose different canonical than user" or "Duplicate without user-selected canonical". When you see those on programmatic pages, Google is telling you two pages are interchangeable. Merge them, or make them differ in substance.

Submission: what it can and cannot do

Method	Scale	Notes
XML sitemaps	Unlimited	The main channel. Discovery only
Request indexing in Search Console	A small daily number per property	For spot fixes, not for bulk
Google Indexing API	200 publish requests per day per Cloud project by default	Officially for JobPosting and BroadcastEvent pages only. Google says submissions go through spam detection and that attempts to get around quotas, including with multiple accounts, may lead to access being revoked
IndexNow	Up to 10,000 URLs per request	Bing, Yandex, Naver, Seznam and others. Not used by Google. See the IndexNow guide

If your pages are job listings or livestreams, the Indexing API is built for you and worth integrating, with URL_DELETED notifications when listings expire. For other content, be realistic: a notification can prompt a crawl, and it does nothing about the quality decision that follows. To see how long it would take to work through a big set at the default quotas, try the Indexing Time Calculator.

Measure by template, not by URL

Looking at individual URLs does not scale. Set up reporting at template level:

1. Indexed share per sitemap. Page indexing report, filtered by each segmented sitemap. Track weekly.
2. Status distribution. A template sitting mostly in "Discovered" has a crawl or linking problem. Mostly "Crawled - currently not indexed" means a quality problem. Mostly duplicates means a differentiation problem.
3. URL Inspection API sampling. The API allows 2,000 inspections per day per property. Sample a few hundred URLs from each template on a rotating basis and store the coverageState and lastCrawlTime. Subfolders added as separate URL-prefix properties each get their own quota. TurboIndex does this automatically for the URLs in your sitemaps, if you would rather not build it.
4. Impressions per template. Performance report filtered by URL pattern, or the BigQuery bulk export. Indexed pages with zero impressions over several months are candidates for improving or pruning.
5. Log files. What Googlebot really fetches, by template. See log file analysis for Googlebot.

Mistakes that sink programmatic projects

• Launching the full set on a new domain with no links and no history.
• Indexable pages for empty or near-empty data.
• Relying on the sitemap alone, with no hub pages.
• Infinite scroll or JavaScript-only pagination on hubs.
• Setting lastmod to the build time on every URL.
• Generating filler paragraphs with an LLM to make pages "unique". The text differs and the information does not, and Google evaluates the information.
• Rotating Google Cloud projects to get past the Indexing API quota. Google's documentation names this explicitly as abuse.
• Never pruning. Pages that have been live for a year with no impressions drag down the template's average. Noindex or remove them and watch the rest.

Pages that get indexed and then fall out again are common in programmatic sets. The causes and the recovery path are covered in why pages get deindexed.