WordPress Indexing Problems: Causes and Fixes

When a WordPress site or a group of its pages is not indexed, the cause is nearly always one of five things: the "Discourage search engines" checkbox, a noindex set in Yoast or Rank Math, a robots.txt rule, a sitemap that is missing or conflicting, or piles of thin archive and attachment pages diluting the site. All of these can be checked in about fifteen minutes, in the order below.

1. The "Discourage search engines" checkbox

Where: WordPress admin > Settings > Reading > Search engine visibility > "Discourage search engines from indexing this site".

Developers tick it during the build and forget it at launch. In current WordPress versions, the setting adds this to every page:

<meta name='robots' content='noindex, nofollow' />

Older versions (before 5.3) wrote Disallow: / into the virtual robots.txt instead. Either way, the whole site is kept out of Google.

How to confirm: view source on the homepage and search for noindex. Or run the URL through the Indexability Checker. In Search Console, affected pages show as "Excluded by 'noindex' tag".

Fix:

1. Untick the box, click Save Changes.
2. Purge all caches: your caching plugin, the host's server cache and the CDN. A cached copy of the page keeps serving the old tag.
3. Reload the source and confirm the tag is gone.
4. In Search Console, inspect the homepage, run Test live URL, then Request indexing. Resubmit the sitemap.

Some hosts and staging plugins set this option automatically on staging copies and carry it over when staging is pushed to production. If it keeps coming back, that is where to look.

2. SEO plugin noindex settings

Yoast SEO and Rank Math can noindex whole content types, taxonomies or single posts. One wrong toggle removes a whole section from Google.

Yoast SEO

• Global, per content type: Yoast SEO > Settings > Content types > (Posts, Pages, your custom types) > "Show [type] in search results". Off means noindex on every item of that type and removal from the sitemap.
• Taxonomies: Yoast SEO > Settings > Categories & tags, same toggle.
• Archives: Yoast SEO > Settings > Advanced > Author archives, Date archives, Format archives.
• Single post: in the editor, Yoast SEO panel > Advanced > "Allow search engines to show this content in search results?". Look for "No".

Rank Math

• Global: Rank Math SEO > Titles & Meta > (Posts, Pages, each custom post type) > Robots Meta. If "No Index" is ticked, the whole type is noindexed.
• Taxonomies: Titles & Meta > Categories / Tags.
• Empty archives: Titles & Meta > Misc Pages > Noindex Empty Category and Tag Archives, which is a sensible option to leave on.
• Single post: editor > Rank Math panel > Advanced tab > Robots Meta.

Menu labels move a little between plugin versions. The principle holds: check the global setting for the content type, then the individual post.

Custom post types

This catches a lot of people. A theme or plugin registers a post type (portfolio, listings, events) and the SEO plugin defaults it to hidden, or the post type is registered with 'public' => false or 'exclude_from_search' => true. Check the SEO plugin's setting for that type, and that the type appears in the sitemap.

Finding noindex in bulk

Crawl the site with a desktop crawler and filter by "noindex", or open Search Console > Indexing > Pages > Excluded by 'noindex' tag and look at the URL patterns. If the list is all /tag/ and /author/ URLs, that is probably intended. If it contains posts or products, something is misconfigured. More in excluded by noindex tag.

3. robots.txt

WordPress serves a virtual robots.txt when no physical file exists. The default is fine:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/wp-sitemap.xml

Problems start when someone edits it (both Yoast and Rank Math have a robots.txt editor under their Tools or General Settings) or uploads a physical robots.txt to the web root, which overrides the virtual one.

Rules that cause damage on WordPress:

Rule	Why it is a problem
Disallow: /	Blocks everything. A staging leftover
Disallow: /wp-content/	Blocks theme CSS, JavaScript and uploaded images, so Google renders broken pages and cannot index images
Disallow: /wp-includes/	Blocks core scripts needed for rendering
Disallow: /*?	Blocks every URL with a query string, including some pagination and assets with version parameters
Disallow: /category/	Hides category pages from crawling, while they often stay indexed without content

Test any URL against the live file with the Robots.txt Tester. Background in the robots.txt guide.

4. Sitemap confusion

WordPress can end up with several sitemaps:

Source	URL
WordPress core (5.5 and later)	/wp-sitemap.xml
Yoast SEO	/sitemap_index.xml
Rank Math	/sitemap_index.xml
Other sitemap plugins	Often /sitemap.xml

Yoast and Rank Math switch off the core sitemap when their own is enabled. Trouble comes from running two SEO plugins or an extra sitemap plugin at once, or from an old sitemap URL still submitted in Search Console and now returning 404.

What to do:

1. Pick one sitemap source. Deactivate the others.
2. Open it in a browser. A 404 on /sitemap_index.xml is usually fixed by Settings > Permalinks > Save Changes (which flushes rewrite rules), or by excluding the sitemap from your caching plugin.
3. Validate it with the XML Sitemap Validator. Look for noindexed URLs, redirects and non-canonical URLs, none of which belong there.
4. In Search Console > Sitemaps, remove old entries and submit the right one.
5. Exclude content types you noindexed. The plugins do this automatically when you use their "show in search results" toggles, which is a reason to use those toggles over custom code.

A blank white page or an XML parsing error on the sitemap is often caused by another plugin or the theme printing whitespace or a PHP notice before the XML. Turn plugins off one at a time to find it.

5. Thin archive and attachment pages

WordPress generates many URLs from one post: the post itself, category archives, tag archives, author archive, date archives, paginated versions of each, and a separate attachment page for every uploaded image. On a blog with 200 posts that easily makes several thousand URLs, most of them thin or duplicated.

Google crawls them, indexes few, and the Page indexing report fills with "Crawled - currently not indexed" and duplicate statuses. On a small site that is mainly noise. On a larger one it slows down the crawling of real content.

Attachment pages

An attachment page is a URL such as /my-post/image-name/ that holds one image and nearly nothing else. Since WordPress 6.4, attachment pages are disabled on new installs and redirect to the file. Sites created earlier still have them unless a plugin handles it.

• Yoast: Settings > Advanced > Media pages. Keep media pages disabled, so that attachment URLs redirect to the media file.
• Rank Math: General Settings > Links > Redirect Attachments, on.

Tag pages

Tags are the worst offender: hundreds of tags with one or two posts each, often identical to a category or to another tag. Options:

• Delete tags you do not need, and redirect the ones that had traffic.
• Noindex the tag taxonomy in your SEO plugin.
• Or treat a small set of tags as real topic pages, with an introduction and enough posts to be useful.

Look at Search Console > Performance, filtered to URLs containing /tag/, before deciding. If they receive no clicks, noindexing is low risk.

Author and date archives

On a single-author blog the author archive duplicates the blog index. Disable it (Yoast: Settings > Advanced > Author archives; Rank Math: Titles & Meta > Authors). Date archives rarely have search value, so disable or noindex them as well.

Other URL noise

• ?replytocom= comment reply links on older themes
• /feed/ URLs (normal, and they appear as "Crawled - currently not indexed", ignore them)
• Internal search URLs /?s= (WordPress noindexes these by default)
• Paginated comments

For the wider picture on low value pages, see thin content and indexing.

Less common causes

Maintenance and "coming soon" plugins. A proper maintenance mode returns a 503. Left on for days, it gets pages dropped. Some "coming soon" plugins return a 200 with a near-empty page, which Google treats as thin or soft 404. Some add noindex.

Security plugins and firewalls blocking Googlebot. Wordfence, host-level WAFs or Cloudflare rules with aggressive rate limiting or country blocking. Googlebot crawls mostly from US IP addresses. Symptoms: "Blocked due to access forbidden (403)" or 5xx spikes in Search Console > Settings > Crawl stats.

Wrong site address. Settings > General > WordPress Address and Site Address set to http:// while the site runs on HTTPS, or with the wrong www variant. That gives mixed canonicals and redirect chains. Both fields should hold the exact canonical origin. Check canonicals with the Canonical Tag Checker.

Password protected and private posts. Private posts return 404 to logged-out visitors, Googlebot included. Password protected posts show only a password form.

Page builders and lazy-loaded content. Content that appears only after a click or scroll event may be absent from Google's rendered HTML. Test with URL Inspection > Test live URL > View tested page.

Hacked sites. Injected spam pages or redirects that only trigger for Googlebot. Check Search Console > Security issues, and look for unfamiliar URLs with a site: search.

A 15 minute diagnostic checklist

• Settings > Reading: "Discourage search engines" is unticked
• Homepage source contains no noindex, and no X-Robots-Tag: noindex header
• /robots.txt has no Disallow: / and does not block /wp-content/ or /wp-includes/
• SEO plugin: posts, pages, products and custom types set to show in search results
• Exactly one sitemap source, which loads, validates and is submitted in Search Console
• Settings > General URLs match the canonical HTTPS origin
• Attachment pages redirect, and thin tag, author and date archives are disabled or noindexed
• No maintenance or coming soon mode active
• Search Console: no manual action, no security issue
• Crawl stats show no 403 or 5xx spikes

After fixing

Purge every cache, check the page source again, then request indexing for the most important URLs and let the sitemap do the rest. Recovery from a site-wide noindex usually begins within days, and the long tail of pages follows over a few weeks.

Both major SEO plugins support IndexNow, which notifies Bing and other participating engines of changes at once. Google does not use IndexNow. For the Google side, TurboIndex can read your WordPress sitemap, submit URLs within Google's quotas and track each URL's index status through the URL Inspection API, which tells you if the fix worked without pasting URLs into Search Console one at a time. For the general process, see how to get Google to index your site.