The Inner Workings of Robots, Spiders, and Web Crawlers | WebReference

The Inner Workings of Robots, Spiders, and Web Crawlers

The Inner Workings of Robots, Spiders, and Web Crawlers

By Lee Underwood

There are three basic types of search engines: crawler-based, human-powered, and a combination of both.

The human-powered search engines - directories - don't really search. They rely on input from other humans. A Web site URL is submitted manually to the directory, sometimes with a short summary of the site. It's reviewed (though not always) by a human and then indexed. Directory-based search engines include Yahoo, Open Directory, and LookSmart.

The indexes of crawler-based search engines are fed data by computer programs called robots. They are also called spiders or Web crawlers because they 'crawl' over the Web. Google, AllTheWeb, and Teoma are all crawler-based search engines.

Search Engine Robots
Search Engine     User-Agent

According to The Web Robots FAQ, "A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced." The robot doesn't physically visit the site, such as a virus would do. It accesses it by requesting documents, much like a Web browser.

There are hundreds, if not thousands, of robots sweeping across the Internet 24/7/365. Their goal is the same: track down information and return it to the sender. Most robots are useful. They feed data to search engines like Google. Many of them are used to map the Internet. Some do a combination of mapping and indexing. A partial list of the robots (user agents) used by search engines is shown at right. A more complete list can be found at the Web Robots Pages.

There are some robots that have a more sinister purpose. Robots like EmailSiphon and Cherry Picker, for instance, are spambots. They're looking to harvest e-mail addresses to add to their spam lists. Those robots scan the page looking for two things:

  1. E-mail addresses, which are then used as targets for spam.
  2. Hyperlinks, which the robots then follow, beginning the process again.

For the most part, robots are pretty straightforward when it comes to traversing a Web site. But robots don't think for themselves; they are programmed to operate in a precise manner. They don't understand JavaScript, frames, or Flash very well. They can't access password-protected areas. Often, robots can't properly index a site because there are too many bells and whistles. If they encounter any roadblocks, they move on. They don't try to figure out how to go around them.

When a robot visits a Web site it does one of two things:

  1. it looks for the robots.txt file and the robots meta tag to see the "rules" that have been set forth, or
  2. it begins to index the Web pages on your site.
Most, but not all, search engine robots follow the rules in the robots.txt file (more on that later). The robot then scans the visible text on the page, the content of various HTML tags, and the hyperlinks listed on the page. It will then analyze the information and process it according to an algorithm devised by its owner. Depending on the search engine, the information is indexed and sent to the search engine's database. Google's process, for instance, takes two steps:

The graphics below show how a robot typically works (courtesy of Kent State University Library).

How robots work - 1

How robots work - 2

You can tell if your site has been visited by robots by viewing your access logs. For a more detailed discussion, see "SpiderSpotting: When a Search Engine, Robot or Crawler Visits." Another good source is "Building a Better Spider-Trap."

If your Web site is online, robots will eventually come to visit. To expedite the process, most of the crawler-based search engines allow links to be submitted manually. It doesn't mean that you will be listed immediately or even the next day; it can take several weeks (or even months). However, it can help to speed-up the process so they will not have to go looking for you.

Created: August 18, 2004
Revised: August 25, 2004