For a page to appear in Google Search, it must be crawlable and indexable. Google's indexing controls guide (opens in a new tab) and your broader robots meta strategy both matter here, because three separate mechanisms can block indexing: noindex meta tags, X-Robots-Tag HTTP headers, and robots.txt rules.

Code Examples#

❌ Avoid — important page accidentally noindexed#

<!-- Product page that should rank in search -->
<meta name="robots" content="noindex, nofollow">
<!-- This page will never appear in Google Search -->

❌ Avoid — Next.js metadata with indexing disabled#

// app/products/[slug]/page.tsx — accidental noindex
export const metadata = {
  robots: {
    index: false,   // This blocks indexing for ALL product pages!
    follow: false,
  },
}

✅ Correct — indexable page (no noindex)#

<!-- Default: no meta robots tag = index, follow -->
<!-- Or be explicit: -->
<meta name="robots" content="index, follow">

✅ Correct — Next.js metadata enabling indexing#

export const metadata = {
  // For pages you want indexed, either omit 'robots' or set explicitly:
  robots: {
    index: true,
    follow: true,
  },
}

✅ Limiting noindex to genuinely private content#

// pages/api/preview-post.ts — only noindex preview/draft pages
export async function generateMetadata({ params }) {
  const post = await getPost(params.slug)
 
  return {
    robots: post.published
      ? { index: true, follow: true }
      : { index: false, follow: false },  // Draft only
  }
}

✅ Testing indexability#

# Check X-Robots-Tag header
curl -I https://yoursite.com/important-page | grep -i 'x-robots'
 
# Check meta robots in HTML
curl -s https://yoursite.com/important-page | grep -i 'robots'

Why It Matters#

• No index = no ranking: A page with noindex cannot appear in search results under any circumstances.
• Common accidents: CMS platforms often default new pages to noindex (e.g., draft mode, staging environments accidentally promoted, category pages in WordPress).
• Silent failure: Indexing blocks cause no visible errors on the frontend. Only Google Search Console (opens in a new tab) or log analysis reliably surfaces the problem, and those same checks help uncover indexability conflicts.

The Three Indexing Mechanisms#

Exceptions#

• Staging, utility, login, account, or internal search pages may intentionally use different crawl or index signals if they are not meant to rank.
• Temporary migration states can produce noisy intermediate signals; flag the live production URL pattern, not one-off transition artifacts.
• When redirects, canonicals, robots directives, or indexability signals conflict, fix the strongest final signal first instead of reporting every downstream symptom as a separate blocker.

Standards#

• Use these references as the standard for the final search-facing HTML, metadata, and crawl behavior.
• Check the implementation against Google Search Central: Prevent Google from indexing pages before treating the rule as satisfied.
• Check the implementation against Google Search Central: Robots meta tag before treating the rule as satisfied.

Verification#

Automated Checks#

• Use Google Search Console Coverage report → "Excluded" tab to find pages marked "Indexed, though blocked by robots.txt" or "Noindex page".
• Use URL Inspection in Search Console for specific URLs.

Manual Checks#

• Export your top 50–100 pages by importance (traffic, revenue, links).
• For each, check: meta robots, X-Robots-Tag header, robots.txt.
• Fix blocks and request re-indexing.