A web crawler is a relatively simple, automated program, or script, that methodically scans or "crawls" through Internet pages to create an index of of the data it's looking for. Alternative names for a web crawler include web spider, web robot, bot, crawler and automatic indexer.
A web crawler can be used for many purposes. Probably the most common use associated with the term is related to search engines. Search engines use web crawlers to collect information about what is out there on public web pages. Their primary purpose is to collect data so that when Internet surfers enter a search term on their site, they can quickly provide the surfer with relevant web sites.
|