What happens when you type a search

When you type a word into Google, Bing, or another search engine and press Enter, you are not searching the entire web in real time. Instead, you are searching an enormous index — a pre-built catalogue of billions of web pages that the search engine has already visited and stored. The search engine looks through that index for pages matching your words, ranks them by relevance, and shows you the results in seconds.

The speed is possible because the work happens before you search. Search engines send out automated programs called crawlers (or spiders) that visit web pages constantly, read their content, and report back. Those crawlers follow links from page to page, discovering new content and revisiting old pages to catch updates. The search engine then processes what the crawlers found and stores it in a way that makes searching fast.

Key Takeaways

  • Search engines use automated crawlers to visit and read web pages, then store that information in an index so they can search it quickly when you type a query.
  • The order of results depends on hundreds of factors including how often your search words appear on the page, whether the page is linked to by other trusted sites, and how recent the page is.
  • Search engines make money primarily through advertising, which is why ads appear at the top of your results and why search engines want to keep you searching.
  • You can see which pages a search engine has indexed by typing "site:example.com" into the search box, and you can ask a search engine not to index your own website.

How crawlers find and read pages

A search engine crawler starts with a list of known web addresses and visits each one. When it reaches a page, it reads the text, images, and links on that page. It then follows those links to new pages and repeats the process. This happens continuously — crawlers are visiting pages right now, and they will visit the same page again weeks or months later to see if anything has changed.

Crawlers are not human readers. They cannot watch videos, run JavaScript code that loads content after the page opens, or see images the way you do. They read the underlying code of the page. If a website is built in a way that hides text inside code that only runs in your browser, the crawler may not see that text at all. This is why some websites appear in search results with little or no description — the crawler could not read the content.

Website owners can control which pages crawlers visit by creating a file called robots.txt in the root folder of their website. This file tells crawlers which pages to visit and which to skip. A page can also include a tag that tells crawlers not to index it, even if the crawler reads it. Search engines respect these instructions, though they are not legally required to.

How search engines rank results

Once a page is in the index, the search engine needs to decide where to rank it when someone searches for a particular word or phrase. Google and other major search engines use ranking algorithms — mathematical formulas that score each page based on hundreds of factors. The pages with the highest scores appear first.

Some factors are obvious. If you search for "how to bake bread," pages that mention "bread" and "bake" many times will rank higher than pages that mention those words once. Pages with those words in the title or heading rank higher than pages with them buried in the middle. But relevance alone does not determine rank.

Search engines also look at links. If many other websites link to a page, the search engine treats that as a sign the page is trustworthy and useful. A link from a well-known, trusted site counts for more than a link from an unknown site. This is why news articles from major outlets often rank high — they get linked to by many other sites. Search engines also consider how recent a page is, whether it loads quickly, whether it works on mobile phones, and whether it has been marked as spam or low-quality by users.

The exact formula is secret. Google has published general information about what matters, but the precise weight given to each factor changes constantly. This is why the same search can show different results on different days, and why search engine optimization (SEO) — the practice of trying to rank higher — is more art than science.

Why search results are not neutral

Search engines make money by showing you advertisements. The ads appear at the top of your results, marked with a small "Ad" label, and sometimes along the side. The search engine charges advertisers each time someone clicks an ad. This creates a built-in conflict: the search engine wants you to keep searching and clicking, not necessarily to find the answer to your question quickly.

This affects what you see in several ways. Search engines may rank their own products higher — Google's own services like Google Maps and Google Shopping often appear prominently in results. They may also rank pages that keep you on the search engine longer, such as pages with lots of ads or pages that are part of larger websites the search engine has decided are authoritative. And because the algorithm is secret, you cannot always tell whether a result is high-ranked because it is genuinely useful or because of other factors.

Different search engines use different algorithms and different sources of information. Bing, DuckDuckGo, and smaller search engines may rank the same search differently. Some search engines, like DuckDuckGo, claim not to track your searches or sell that information to advertisers, while Google and Bing use your search history to personalize results.

What search engines store about you

When you search on Google or Bing while logged into your account, the search engine stores your search history. It records what you searched for, when you searched, and which results you clicked on. This information is tied to your account and your device. Search engines use this data to personalize your results — showing you pages they think you will find more relevant based on your past searches — and to build a profile of your interests for advertisers.

If you search while not logged in, or in a private browsing window, the search engine still knows your search happened (it can see your IP address), but it may not tie it to your account. Different search engines have different policies about how long they keep this data and what they do with it. You can view and delete your search history on Google and Bing by going to your account settings, though deletion does not necessarily remove the data from the search engine's servers.

How to see what a search engine knows about a page

You can find out whether a search engine has indexed a specific page by typing site:example.com into the search box, replacing "example.com" with the actual website address. This shows you all the pages from that site that the search engine has indexed. If a page you expect to see is missing, it may not have been crawled yet, or the website owner may have asked the search engine not to index it.

You can also see how a search engine sees a page by using Google's Cache feature. When you search and see a result, click the small arrow next to the page title and select "Cached." This shows you the version of the page that Google stored in its index, which may be older than the current version of the page. This is useful if a website is down or has changed significantly.

If you own a website, you can tell search engines how to crawl it by creating a sitemap — a file that lists all your pages and tells crawlers which ones to visit first. You can also use Google Search Console or Bing Webmaster Tools to see which searches bring people to your site, which pages rank, and whether the search engine has found any problems crawling your pages.

Why some pages do not appear in search results

Not every page on the web appears in search results. Some pages are deliberately hidden. Website owners can use robots.txt or a meta tag to tell crawlers not to index a page. Pages behind a login wall — like your email inbox or your bank account — are not indexed because crawlers cannot log in. Pages that are very new may not appear yet because crawlers have not visited them.

Some pages are removed from search results because they violate the search engine's policies. Google and Bing remove pages that contain malware, pages that are used for phishing, and pages that have been reported as spam by many users. If a page contains personal information like your home address or phone number, you can ask Google to remove it from search results, though the page itself remains on the web.

Pages can also fail to rank well because they are hard for crawlers to read. Pages built entirely in Flash, pages that load content through JavaScript, and pages with very little text may not be indexed properly. This is why modern web design emphasizes clean code and text content — it helps both crawlers and human readers.

Frequently Asked Questions

Can I remove my website from search results?

Yes. You can create a robots.txt file that tells crawlers not to visit your pages, or add a meta tag to each page telling crawlers not to index it. You can also use Google Search Console to request that specific pages be removed. However, if your site is linked to by other websites, people may still find it through those links even if it does not appear in search results.

Why do I see different results on my phone than on my computer?

Search engines rank pages differently depending on the device you are using. Pages that work well on mobile phones rank higher when you search on a phone. Search engines also personalize results based on your location and your search history, which may differ between devices if you are logged into different accounts or if one device has a different location history.

How long does it take for a new page to appear in search results?

It depends on how discoverable the page is. If other websites link to your new page, crawlers may find it within days. If your page is only linked from your home page, it may take weeks or months for crawlers to visit it. You can speed this up by submitting your page to Google Search Console or by creating a sitemap.

Do search engines read the content inside images?

Search engines cannot read text inside images the way they read text on a page. However, they can read the filename of the image and the "alt text" — a description you provide for accessibility. If you want an image to be searchable, use a descriptive filename and add alt text that describes what the image shows.

Why does the same search show different results for different people?

Search engines personalize results based on your location, your device, your search history, and your account settings. They also test different ranking algorithms with different users to see which one works better. This means two people searching for the same thing at the same time may see different results, which is why it is hard to know what "the" top result for a search actually is.