Search Engines

There are many different types of search engines presently being used on the Internet to find information. The bulk of these fall into two main categories: Robot-Generated Web Indexes and Subject Directories. Generally speaking, most people end up using a "true" search engine or one that is really a robot-generated index where there is no human involvement in its creation.

Robots (or spiders, intelligent agents, wanderers, or crawlers as they might otherwise be called) are computer programs that are designed to explore and collect information about Universal Resource Locators (URLs) and subject matter on the World Wide Web (WWW). They compile their findings into a database, and then generate an internal index that allows users to search. So when you search by typing a word or by entering your question into the text box, that word or query is searched against the internal index or database. Examples of this type of search engine include Google, Teoma, and AllTheWeb.

By comparison, a subject directory is not built by robots and consists of a "browsable" index of categories and subcategories organized by subject. A person actually determines where a particular URL (e.g., http://www.queenslibrary.org) should be placed. Note that while many of these directories now include the ability to keyword search like a search engine, there continues to remain a section that is strictly organized by humans. Example of these subject directories include Yahoo and Open Directory.

A third category that should be noted is that of Metasearch Engines. Searches with this type of engine lead to several search engines and/or directories being searched at once. The findings or results are combined on to one page or folders, conveniently allowing the searcher to view multiple results quickly. Of course, one benefit of this process is that time is saved. Why search for results at individual sites when you can do it all at one site? A list of Metasearch Engines can be found on SearchEngineWatch.


Why Do I Get Different Results?

Robots methods of gathering information depend on proprietary standards. For this reason, the databases that are created to support an index vary both in structure and in organization. Consequently, a search for "vacation sites" in Alta Vista will generate a different search result set in Ask. So as a rule, be aware that all search engines are not created equal.


To summarize, here's a quick guide to differentiate between Robot-Generated Web Indexes, Subject Directories, and Metasearch Engines.

Robot-Generated Web Indexes, possible to search by two methods:

  • Keyword-only
    Enter a word or phrase into a search dialog box
  • Keyword and subject directory searching
    Search by keyword and also be able to browse a subject directory or catalogue

Subject Directories

  • Built by humans
  • Consists of a browsable index of categories and subcategories organized by subject
  • Generally includes the ability to keyword search within the directory.

Metasearch Engines

  • Neither a robot-generated web index or a subject directory
  • Searches several search tools at once
  • Combines results on to one page

For more information about the topic of search engines, have a look at the following sites:

SearchEngine Watch

Scout ToolKit: Searching the Internet