How Search Engine Works
This article aims to explain briefly, concisely and fully how search engine works. The article consists of the following topics:
• Types of search engines
• How does a search engine work?
• Natural & Sponsored search results
• Search engine marketing
• Search engine optimization (SEO)
• Search engine advertising (SEA)
Types of search engines
There are different types of search engines, namely a crawler search engine, a 'subject directory' search engine and a 'meta' search engine.
Crawler search engine
A search engine based on a crawler stores a copy of the world wide web in a database, the index. After a search, the search results are ranked by relevance from the index.
The best known and most widely used crawler search engine is Google .
Subject directory
'Subject directories' are online guides with a collection of links per topic that are tracked manually. The best-known example in the Netherlands is Startpagina.nl .
Vertical search engines
Special search engines, better known as vertical search engines, also fall under 'subject directories'. A vertical search engine is a search engine that is delineated for a
specific type of information or a specific topic or interest. A vertical search engine often makes the results more relevant. An example of a vertical search engine is Google
Maps (geographically delineated).
Meta search engine
A meta search engine uses multiple search engines as a basis and combines the search results of multiple search engines. An example of a meta search engine in the Netherlands
is Vinden.nl .
How does a crawler search engine work?
The most used search engines, with Google in the lead, are a crawler search engine, which will further deepen this type of search engine.
For the sake of convenience, I mean from now on with 'search engine' a search engine based on a crawler.
A search engine based on a crawler consists of:
• Crawler (spider, robot)
• Index (database)
• Algorithm (formula)
The crawler
The crawler, also known as 'spider' or '(ro) bot', is a relatively simple computer program that does nothing but follow links and store the HTML version of the web page in a
database, the index. The HTML version of a web page is the page you see with a web browser (such as Internet Explorer).
A crawler can only follow the links on web pages that are already included in the index. Without a link to a web page (or without logging in a web page with a search engine)
a webpage can never be found!
Sign in to search engines
With a new or not frequently visited website or web page it may take weeks before a search engine finds the website, because there are no or few links to the relevant website
or web page.
To this end it is possible to register your website with the search engines. Because Google is by far the most used search engine in the Netherlands, logging in with Google
is recommended. But getting better and faster is getting links to your website from already indexed web pages.
For example, registering with Google does not guarantee that the website will also end up at Google. That determines the algorithm of a search engine, which is explained
below.
The index
The crawler or spider stores the found HTML version of a web page in a gigantic database, the index. At that time a web page has been indexed. Primarily, only the text on a
web page is stored in the index, but the search engines keep storing more and more information.
This index is comparable with the index in the back of the book. The index 'knows' which words are on which page. Next, it is up to the search engine algorithm to determine
in which order the search results are ranked.
The algorithm
The algorithm of a search engine is a complex mathematical formula with the important task of determining which web pages from the index are shown in a search.
The main goal of a search engine is to provide the most relevant search results based on a search. This very complex task lies in the algorithm of search engines.
How does the algorithm of search engines work?
The algorithm of a search engine is a mathematical formula that uses hundreds of factors to determine the relevance of a web page to a search.
The specific operation of such algorithms is secret, because the success of a search engine depends on the relevance of the search results. Many important tests and
experiments have revealed a number of important factors, but the precise operation is unknown.
Google PageRank
The most famous piece of an algorithm of a search engine is Google's PageRank technology. The Google PageRank is popular and is often seen as the algorithm of Google, but it
is just one of the many factors that Google uses within the algorithm.
Google explains PageRank itself as: Instead of counting direct links, Google interprets a link from page A to page B as a 'vote' from page A on page B. Google assesses the
relevance of a page by counting the received votes .
The Google PageRank technology also takes into account the importance of the webpage that issues a 'voice'. A voice of a page that is important itself weighs heavier and
helps to make the page for which a vote has been made more important.
Also read my extensive Google PageRank manual .
The algorithm is of course continuously updated and extended. In 2013, Google completely renewed the complete algorithm. The new algorithm is called Hummingbird .
Natural and sponsored search results
Most search engines provide natural search results and sponsored search results.
The natural search results are the 'head' search results of a search engine on the left of a search engine. The natural search results are ranked on the basis of the
algorithm described above, of course after indexing by a crawler.
The sponsored search results are the smaller search results that are shown on the right or the top of the search results. The sponsored search results are primarily ranked
based on an auction system, with advertisers paying a self-defined amount per click on the ad.
Secondarily, the sponsored search results are ranked based on an increasingly complex algorithm, which determines the relevance of the advertisement relative to the keyword
on the one hand and landing page of the advertisement on the other hand.
Read more about the Google advertising program in my AdWords manual .
Search engine marketing
The commercial use of search services, which usually involve search engines, is known as search engine marketing. Search engine marketing consists of search engine
optimization (SEO) and search engine advertising (SEA), in short, influencing the natural and sponsored search results.
Search engine optimisation
Search engine Optimization is actively influencing the unpaid search results. This mainly concerns unpaid search results in which texts of web pages are searched, but also
think of mobile web pages, images, videos and all other current and future forms of information.
Search engine optimization has three optimization areas, namely the Text factor, the Indexation factor and the Popularity factor. The text factor and indexation factor are
mainly factors within the website, while the popularity factor is mainly influenced on factors outside the website.
Text factor
The text factor consists in particular of examining the right words and texts. The choice of (search) words and their placement on the web page is important here. The goal is
to select words and texts that the target group of the website uses to search for what the webpage offers.
The text factor is important for a search engine to determine what the web page is about. On this basis, the search engine can determine the relevance of that web page
compared to a search.
Indexation factor
The indexation factor concerns the extent to which a crawler can "read" the web page. As described, a crawler reads the HTML code. Other techniques for presenting a website,
for example with Flash or Javascript, can make a web page unreadable and therefore untraceable for a search engine.
Popularity factor
The popularity factor mainly concerns external factors of a web page. The popularity factor is mainly concerned with relevant and qualitative links to the relevant webpage,
because the algorithm of search engines attaches great importance to this.
More information and articles about search engine optimization can be found on this weblog in the search engine optimizationcategory .
Search engine advertising (SEA)
Search engine Advertising is simply buying advertising space in and around search services, where advertising in search engines, with Google AdWords as the most famous
advertising platform.
The main feature of Search Engine Advertising is that the ads are linked to a search (indicating the user's intent), and are therefore relevant to the search.