This class implements a user agent that is suitable for robot applications. Robots should be nice to the servers they visit. They should consult the /robots.txt file to ensure that they are welcomed and they should not make requests too frequently. B...

This class implements a user agent that is suitable for robot applications. Robots should be nice to the servers they visit. They should consult the /robots.txt file to ensure that they are welcomed and they should not make requests too frequently. B...

NOTE: You can find a full description of the Scraper framework in WWW::Scraper::ScraperPOD.pm. "Scraper" is a framework for issuing queries to a search engine, and scraping the data from the resultant multi-page responses, and the associated detail p...

RobotPUA is a simple frontend to the LWP::Parallel::RobotUA module. It is here in order to maintain the compatibility with previous releases. However, in order to prevent the previous need for changing the original LWP sources, all extension files ha...

The libwww-perl collection is a set of Perl modules which provides a simple and consistent application programming interface (API) to the World-Wide Web. The main focus of the library is to provide classes and functions that allow you to write WWW cl...

Introduction ParallelUserAgent is an extension to the existing libwww module. It allows you to take a list of URLs (it currently supports HTTP, FTP, and FILE URLs. HTTPS might work, too) and connect to all of them _in parallel_, then wait for the res...

Just like a RobotUA, but you can see how long it will wait for a given link and then try another....

This module implements a user agent which remembers where it has been and when so that the user can avoid too fast visits, but doesn't actual implement that wait....