Indexing is the process a search engine uses to analyze a web page and store it in its database, called the index, so the page can be shown in search results. If a page is not indexed, it cannot rank, no matter how good it is.
Indexing sits in the middle of how search works. A crawler first finds and fetches a page, then the search engine indexes it, and only then can it serve that page for relevant queries. Increasingly, the same index also feeds AI-generated answers.
What is indexing?
Google's overview of how Search works describes three stages: crawling, indexing and serving results. During indexing, Google tries to understand what a page is about. It processes the text, key tags and attributes like the title element and image alt text, and images and videos.
Google also checks whether the page duplicates something else. It groups similar pages into clusters and chooses one as the canonical, the version most likely to be shown. It then collects signals about that canonical page, such as its language, the country it is relevant to and its usability. All of this is stored in the Google index, which Google describes as a large database hosted on thousands of computers.
One point surprises many site owners: indexing is not guaranteed. Google says plainly that not every page it processes will be indexed.
Why indexing matters
Indexing is the entry ticket to organic search. A page that is not in the index cannot appear on the SERP, cannot earn clicks and cannot build visibility over time.
It also matters for AI search. Google's documentation on AI features in Search says a page must be indexed and eligible to show with a snippet to appear as a supporting link in AI Overviews or AI Mode. Google adds that there are no extra technical requirements beyond that. In other words, getting indexed properly is the foundation for being cited in Google's AI answers.
Other AI assistants use their own crawlers and search partners, and their behavior changes often. As of 2026, the safest assumption is that pages which are crawlable, indexable and clearly written have the best chance across systems.
How indexing works
Discovery and crawling
Before a page can be indexed, it has to be found. Google discovers URLs mainly by following links from pages it already knows and by reading sitemaps you submit. Googlebot then fetches the page. Google's Googlebot documentation notes that it primarily indexes the mobile version of most sites, and that it processes only the first 2 MB of a supported file type, such as an HTML page.
On very large sites, how much Google crawls is shaped by crawl budget. For most sites, discovery is the bigger issue: pages with no internal links pointing to them are easy to miss.
Rendering
Google renders pages and runs JavaScript, similar to a browser. That matters because content that only appears after scripts run must be rendered before Google can see it. Server-rendered content is usually the most reliable to index.
Analysis and canonicalization
Google reads the content and structure, figures out the topic and checks for duplicates. If several URLs show the same content, it picks one canonical URL. You can suggest your preference with a canonical tag, but Google treats that as a hint.
Storage and serving
Google stores the canonical page and its signals in the index. When someone searches, Google looks through the index for matching pages and returns the ones it considers most relevant and useful, using hundreds of factors including location, language and device.
Indexing in practice
Picture a generic accounting firm that launches ten new service pages. A month later, only four show up when the team checks Search Console.
The URL Inspection tool and the Page Indexing report explain why. Two pages show "Excluded by noindex tag" because a staging setting was never removed. Three show "Discovered - currently not indexed" because no other page on the site links to them. One shows "Duplicate, Google chose different canonical than user" because it is almost identical to an older page.
The fixes are specific. The team removes the noindex rule, adds links to the new pages from the services hub and relevant blog posts, and rewrites the near-duplicate so it offers distinct information, or merges it into the older page. Then they request indexing for the key URLs. There is no fixed timeline, but those changes address each actual cause.
How to get pages indexed
- Make sure Google can crawl the page. Do not block important pages in robots.txt, and avoid putting key content behind logins.
- Remove accidental noindex rules. Check both the robots meta tag and the X-Robots-Tag HTTP header.
- Link to every important page. Use descriptive internal links from relevant, already-indexed pages.
- Submit an XML sitemap. List canonical URLs only and keep it current.
- Return correct status codes. Pages that should exist return 200. Removed pages return 404 or 410. Avoid soft 404s.
- Consolidate duplicates. Use canonical tags and redirects so Google has one clear URL per piece of content.
- Publish genuinely useful content. Google lists low content quality as a common reason pages are not indexed.
- Use URL Inspection for priority pages. It shows whether a URL is indexed, which canonical Google chose, and lets you request a recrawl.
- Consider IndexNow for other engines. IndexNow lets you notify participating search engines, such as Bing and Yandex, when URLs change. Google is not listed as a participant, and the protocol notes that submission does not guarantee indexing.
Common indexing mistakes and misconceptions
- Thinking every page should be indexed. Google's Page Indexing report notes that non-indexed URLs can be fine. Duplicates, redirects and intentionally excluded pages should stay out. The goal is to get the canonical version of every important page indexed.
- Using robots.txt to keep pages out of search. Google's noindex documentation explains that a page blocked by robots.txt can still be indexed if other pages link to it. To keep a page out, use noindex and leave it crawlable so Google can see the rule.
- Leaving staging settings live. A site-wide noindex left over from development is one of the most common and damaging launch mistakes.
- Repeatedly requesting indexing. Requesting the same URL over and over does not speed things up. Fix the underlying cause first.
- Confusing indexed with ranking. An indexed page can still get little traffic if it is not relevant or helpful for the queries people actually use.
- Hiding content behind JavaScript. If key text only appears after complex client-side scripts, Google may index it late or incompletely.
Crawling vs. indexing
Crawling means fetching a page. Indexing means analyzing and storing it. Search Console reflects the difference in two statuses. "Discovered - currently not indexed" means Google knows the URL but has not crawled it yet. "Crawled - currently not indexed" means Google fetched it but decided not to add it to the index, at least for now. The first often points to discovery or crawl capacity issues. The second often points to quality or duplication.
How Butter approaches indexing
At Butter, indexing strategy is part of every GEO plan, because pages that are not indexed cannot be cited in AI answers or shown in search. Each month, we review Search Console indexing data for your important pages, look at which canonical Google chose, and flag pages that are excluded for the wrong reasons.
That work turns into specific on-page and off-page recommendations, such as internal links to orphaned pages, cleaning up stray noindex tags, or consolidating thin duplicates. On the GEO+SEO plan, we also handle technical fixes like crawl issues and broken links directly, and the Butter Analytics dashboard brings Search Console data into one place. You can see what each plan includes on our GEO services page.
We cannot make Google index a page, and nobody honestly can. What we can do is remove the obstacles and make each page worth indexing.
Frequently asked questions
How long does it take Google to index a new page?
Google does not give a fixed timeline. It depends on how easily the page is discovered, how often your site is crawled and whether Google sees the page as worth indexing. Internal links, a current sitemap and the URL Inspection tool can help.
How do I check if a page is indexed?
Use the URL Inspection tool in Google Search Console for a definitive answer on a specific URL. The Page Indexing report shows site-wide totals and the reasons pages were excluded.
Why is my page crawled but not indexed?
Google fetched the page but chose not to add it to the index, at least for now. Common causes include thin or duplicate content and weak internal linking. Improving the page and linking to it from relevant pages is usually more effective than resubmitting it.
Does being indexed help with AI Overviews?
Yes, it is a prerequisite. Google says a page must be indexed and eligible to show with a snippet to appear as a supporting link in AI Overviews or AI Mode. Being indexed does not guarantee you will be cited.