Crawler

a.k.a. bot -or- spider -or- wanderer

Synonymous with spider, this is a program that searches the Internet in order to locate new, publicly accessible resources, such as Web pages, files available in public FTP archives, and Gopher documents. Also called wanderers or bots, crawlers contribute their discoveries to a database that Internet users can search by using a search engine.

This type of technology is necessary because the rate at which people create new Internet documents greatly exceeds any manual indexing capacity (which is how it's done with search directories).