There are literally millions of pages full of information, images, files and other data on the World Wide Web. The problem that arose was that all that information was largely unavailable to the millions of Internet users who were not aware of a website’s precise address (url). Thus the search engine was developed with the express job of searching for information uploaded onto the World Wide Web.

While Web Directories are solely operated by human input, search engines operate algorithmically or through a mixture of algorithmic and human input.

• Web Crawling and Indexing

Search Engines make use of a number of web crawlers, often also called spiders, to retrieve information about web pages, for storage. These crawlers are automated software agents that visit websites, read through the information presented therein, read the meta tags provided, and follow the links that the website connects to. Data is extracted from titles, headings and also meta tags. All the information collected by the spider is relayed to a central depository where the data is indexed. From time to time, the crawler will return to sites it has already been to collect any information that has changed from its last run-through. The frequency of this is different between each search engine and is decided by the search engine’s administrator.

The information stored in the indexed databases is utilised to respond to search queries. Some search engines, such as Google, store all or part of a source page (cache) plus information about web pages, while other search engines, such as Alta Vista, store each and every word of every page they visit.

• Querying Search Engines

When searching the web for particular information without having a specific web site in mind, you would generally place your query through a search engine, through the use of phrases or keywords that you envision will result in a list of websites that will provide the information and data you require. The search engine then analyzes its index and provides a list of the best-matching web pages. This list usually includes a short summary with the selected page’s title, and parts of the page’s text that match your search criteria.

• Ranking Search Results

Since web pages run into millions, as do search queries, a search engine will have located a huge amount of web pages containing the search criteria you would have requested. For this reason, search engines employ ranking methods according to which search results are ranked. This is done with the aim of providing relevant results, best link first. Each search engine has its own ways of determining which pages would provide best matches to particular search queries and which order these websites should be ranked in. Same queries placed on different search engines will therefore most probably produce different ranked results. These ranking methods also undergo regular developments and evolutions in line with the ever-changing characteristics of Internet users and technology.

Common ranking methods involve the analysis of web pages for frequency and location of keywords, with those pages with higher frequency of keywords being considered the most relevant. One has to be careful to balance keyword usage with overall text sense to avoid the risk of keyword stuffing and spamdexing that will be penalised by search engines. Another common analysis undertaken by search engine algorithms is the way pages link to other pages in the web. This allows a search engine to determine what web pages are about and whether such pages deserve better results ranking positions. Here too, search engines have become sophisticated enough to locate and overlook artificial links built into sites with the intention of artificially boosting search engine ranking.

• Search Engine Advertising

Since search engines are commercial ventures they must financially support themselves. This is generally achieved through search engine advertising that involves the appearance of adverts alongside the naturally ranked list brought up when users search utilising the same keywords specified by the advertiser. Search Engines earn money each time such an advert is clicked upon.