How do search engines work?

‍When you enter a search query, your search engine is not actively searching web pages for relevant content in real time. What its actually doing is returning results from an already-existing database that has cataloged pages based on hundreds of different criteria.

To build this database, search engines use an automated program to crawl the web. Crawling, in this sense, refers to scanning a page to record things like word counts and hyperlinks. The crawler then follows each of these links to other pages. The crawler, or spider, continuously navigates the entire world wide web of known pages and adds it to the database. Tip: when launching a web page, you can submit it to a search engine rather than waiting for the crawler to find it

How can results be returned so fast?

‍The database is massive, but in order to complete your query, the search engine does not have to load entire web pages --- it only needs to scan through data about pages.

Think of it as a multi-step process of elimination. All pages not containing the search term are eliminated from consideration. This is going to be most of the pages on the world wide web, so this greatly narrows down the portion of the database that the search engine has to deal with. Further narrowing down is done with other criteria, much of which the search engine can infer based on user habits.

Simply mentioning a word or phrase many times on a page does not necessarily make it a relevant web page to the user. There are further categorizations that need to happen, so factors like the user’s search history and geographic location are considered

The continual evolution of search engines

‍Even as we learn how a search engine operates, things are changing or being added in. The search engine algorithm is constantly being improved, or evolving to keep up with rapidly evolving user interests. For an example, lets examine a hypothetical user search for the word sushi in the year 1999.Google had just been released, about a year prior. Consider that:

-the internet was still a novelty to the consumer-most businesses didn’t have a web page

-sushi was less accessible in the United States than it is today

-search engines couldn’t access location services that modern devices like smart phones offer

If you were searching sushi back then, you might have wanted to know what the heck sushi is. In 2020,if you’re searching sushi, you probably want to find a restaurant. You won’t even have to add near me to the query, because the algorithm predicts that you’re looking to get some sushi, and will provide suggestions for restaurants close by.

The people designing the search engine have (hopefully) anticipated the intent of their users. How? Well, its more data than it is divination. This, in itself, in an important aspect of the internet

A new wealth of insight into what the customer wants

‍The internet provides data about the user to the programmers who design the experience for the user. Thus, the programmers always have insight into not only what the user wants, but also into the general direction that user desires seem to be trending. There are many ways that the system collects feedback about the user experience.

For example, if there is a consistent trend of users who search a certain word/phrase and then only click results at the bottom of the first page, the algorithm will re-order the results on the front page.

In some ways, this is just as important as the features that the search engine (or other online resource)provides for the user. This is an unprecedented and powerful tool that automates the collection of data about what the customer wants.

Collecting data about the success of the search engine’s predictions is a huge part of what allows the programmers to keep fine tuning the search engine to bring you more relevant results (at least, more relevant by their definition). Our behavior shapes the search engine. And the search engine’s behavior, in turn, shapes how the user views the world.