How Search Engines Work: Crawling, Indexing, Ranking and AI

Updated 7 October 2026: Clarified crawling versus indexing, robots.txt versus noindex, canonical URLs, and how AI search differs from model training. Added a practical page-diagnosis workflow and current Google documentation.

Search engines discover URLs, crawl their content, decide which pages to index, and select results for each search. These are separate decisions. A page can be accessible but not crawled, crawled but not indexed, or indexed without receiving useful rankings.

If your page is missing from Google, identify which stage needs attention before changing titles or publishing more content. Google's explanation of how Search works is the starting point: meeting its requirements does not guarantee crawling, indexing, or visibility.

1. Discovery and crawling: can Google find and fetch the page?

Google discovers URLs through links from known pages and through sitemaps. Discovery means Google knows a URL exists; crawling means Googlebot has actually fetched it. A sitemap helps discovery but does not force a crawl.

Give every important page a normal, crawlable internal link from a relevant page. For example, a guide explaining search-engine fundamentals belongs in a beginner's reading path, with links from related Search Console and SEO guides. An isolated URL in a sitemap is a weaker reader journey than a page connected to useful content.

When a page is not being fetched, check the response status, whether login is required, and whether robots.txt prevents access. A public article should normally return HTTP 200 and show its main content without requiring account access. Repeated server errors can slow crawling; a historical error report does not by itself prove the server is failing today.

What about crawl budget?

Crawl capacity and demand influence which URLs Google fetches. They are not a fixed page allowance you can increase by publishing more often. Google's crawl-budget guidance primarily concerns large sites. For a small publication, first check discovery, accessibility, duplicate URLs, and useful content rather than assuming crawl budget explains an unindexed article.

2. Indexing: does Google choose to keep this page?

After crawling, Google processes the content and decides whether to index it. Crawlability is necessary for Google to read a page, but an accessible page is not automatically indexed. Similar URLs may be grouped together and represented by one canonical version.

Robots.txt and noindex do different jobs

Robots.txt controls crawling. It is not a reliable way to prevent a known URL from appearing in search: Google may know about the URL from other links without being able to read its content. A noindex directive asks Google not to index a page, and Google must be allowed to crawl that page to see the directive. See Google's noindex instructions.

Canonical URLs clarify duplicate versions

A canonical link identifies your preferred version when multiple URLs show the same content. For example, a product homepage with a referral parameter can declare the clean homepage as canonical. Use that clean URL in the sitemap and ordinary internal links. Canonicalization is a signal; Google can choose a different canonical. It is not a guarantee of indexing. Google's canonical guidance explains the available methods.

Structured data can make eligible content easier to interpret for supported search features. It does not rescue an inaccessible page or guarantee indexing. Likewise, an excluded tag archive is a different situation from an excluded original article: the aim is to index important canonical content, not every URL a site produces.

3. Ranking: does the indexed page answer this search?

Once indexed, a page can compete for relevant searches. Its visibility depends on the query, the usefulness of its answer, competing results, and the signals Google's systems use. A page ranking well for one search may be much less relevant to another.

Start with the reader's task. Explain the answer clearly, include examples that resolve practical decisions, and link to evidence for claims that need support. Google's ranking-systems guide describes multiple systems rather than one public ranking score.

Use analytics to understand whether visitors find your content useful, but do not treat your site's bounce rate or time-on-page figures as a direct Google ranking formula. A third-party Domain Authority score is also not a Google metric. Improving an arbitrary score is less useful than improving the page and the reader journey.

In Search Console, examine the page's actual queries and positions before interpreting CTR. More impressions at low positions may produce few clicks. For a practical workflow, use the guide to finding SEO quick wins.

4. AI search: retrieval and training are different

An AI model's training process is not the same as a search engine's live index. Some assistants retrieve current web pages when answering; others may answer from model knowledge. Do not assume a recent update is present in a model's training data or that a strong organic ranking guarantees an AI citation.

For Google's AI Overviews and AI Mode, Google's website guidance says established SEO practices remain relevant. A supporting page needs to be indexed and eligible to show with a snippet. There is no special AI markup required to qualify, and eligibility does not guarantee inclusion.

Keep content accessible, clear, accurate, and worth referencing. Measure visibility and traffic separately; appearing in an AI answer does not prove someone visited the site. The AI traffic versus AI visibility guide explains that measurement distinction.

5. A practical diagnosis for a missing page

1. Inspect the exact canonical URL. Use URL Inspection rather than relying only on the aggregate Page indexing report. Start with the Search Console setup and first reports guide if needed.
2. Read the indexed status. “Discovered – currently not indexed” means Google knows the URL but has not crawled it. “Crawled – currently not indexed” means it fetched the page without indexing it. A duplicate may correctly point to another canonical URL.
3. Run a live test. Check fetch success, crawl permission, indexing permission, and the declared canonical. A successful live test establishes current eligibility, not that the page is already indexed.
4. Fix the demonstrated problem. Correct an accidental noindex, blocked crawl, server failure, conflicting canonical, or weak discovery path. If access is healthy, improve the page's usefulness and relevant internal links rather than inventing a technical fault.
5. Request indexing once when warranted. After a meaningful correction, submit the eligible URL. Keep the sitemap current and monitor the indexed view. Repeated submissions do not accelerate crawling.

As a hypothetical example, a new public tutorial may return HTTP 200 and appear in the sitemap while still being “Discovered – currently not indexed.” Add a useful link from an established related guide, verify the live test, and submit it once. Do not add more schema or remove the correct canonical merely to change the report.

Google says recrawling can take days to weeks and an indexing request does not guarantee inclusion. Record what you changed and the exact response, then distinguish a queued request from an indexed result. For a reason-by-reason decision and a copyable one-URL record, follow the Search Console page-indexing decision tree.

Where to go next

For the place of organic search within a wider marketing strategy, read SEO vs SEM vs digital marketing. To turn real query groups into priorities for existing pages, follow the Search Console content-map workflow.