Ever wondered about the difference between a search engine and a directory? The term "search engine" is often used to describe three types of web indexers: those that are powered by crawlers, or spiders (not real ones!), those that are powered by human submissions and those that are a combination of the two.
- Crawler-based engines
- A crawler-based engine (like Google) is an automated search facility where programs called spiders "crawl" through sites, scan the code behind the site and index pages based on the information they gather. The crawler will periodically return to the sites to check for any new information. The administrators of the search engine determine the frequency at which this happens. No humans are involved in this process, which is the major difference between a search engine and a directory.
- Dmoz is an example of a directory where people (rather than spiders) review and index information from websites. To be listed with a directory, you submit your details and a short description of your website. The directory's editors then review the site and classify it. They will also decide what category to put it under and where to rank it in their results. A search looks for matches only in the descriptions submitted.
- "Hybrid Search Engines" Or Mixed Results
- In the web's early days, a search engine presented either crawler-based results or human-powered listings. Today, it's extremely common for both types of results to be presented. Usually, a hybrid search engine will favour one type of listing over another. However, it does also present crawler-based results, especially for more obscure queries.