Wednesday, November 15 2006 @ 02:00 PM EST Contributed by: sde
Web spiders are software agents that traverse the Internet gathering, filtering, and potentially aggregating information for a user. Using common scripting languages and their collection of Web modules, you can easily develop Web spiders. This article shows you how to build spiders and scrapers for Linux® to crawl a Web site and gather information, stock data, in this case
A spider is a program that crawls the Internet in a specific way for a specific purpose. The purpose could be to gather information or to understand the structure and validity of a Web site. Spiders are the basis for modern search engines, such as Google and AltaVista. These spiders automatically retrieve data from the Web and pass it on to other applications that index the contents of the Web site for the best set of search terms.
Similar to a spider, but with more interesting legal questions, is the Web scraper. A scraper is a type of spider that targets specific content from the Web, such as the cost of products or services. One use of the scraper is for competitive pricing, to identify the price of a given product to tailor your price or advertise it accordingly. A scraper can also aggregate data from a number of Web sources and provide that information to a user.
