LWP::RobotUA - a class for well-behaved Web robots
use LWP::RobotUA; my $ua = LWP::RobotUA->new( 'my-robot/0.1', 'me@foo.com' ); $ua->delay(10); # be very nice -- max one hit every ten minutes! ... # Then just use it just like a normal LWP::UserAgent: my $response = $ua->get('http://whatever.int/...'); ...
This class implements a user agent that is suitable for robot applications. Robots should be nice to the servers they visit. They should consult the /robots.txt file to ensure that they are welcomed and they should not make requests too frequently.
But before you consider writing a robot, take a look at <URL:http://www.robotstxt.org/>.
When you use a LWP::RobotUA object as your user agent, then you do not really have to think about these things yourself; robots.txt files are automatically consulted and obeyed, the server isn't queried too rapidly, and so on. Just send requests as you do when you are using a normal LWP::UserAgent object (using $ua->get(...), $ua->head(...), $ua->request(...), etc.), and this special agent will make sure you are nice.
robots.txt
$ua->get(...)
$ua->head(...)
$ua->request(...)
The LWP::RobotUA is a sub-class of LWP::UserAgent and implements the same methods. In addition the following methods are provided:
Your robot's name and the mail address of the human responsible for the robot (i.e. you) are required by the constructor.
Optionally it allows you to specify the WWW::RobotRules object to use. If you don't provide one, then this user agent will make its own internal database of robots.txt rules as needed.
Set the minimum delay between requests to the same server, in minutes. The default is 1 minute. Note that this number doesn't have to be an integer; for example, this sets the delay to 10 seconds:
$ua->delay(10/60);
Get/set a value indicating whether the UA should sleep() if requests arrive too fast (before $ua->delay minutes has passed). The default is TRUE. If this value is FALSE then an internal SERVICE_UNAVAILABLE response will be generated. It will have an Retry-After header that indicates when it is OK to send another request to this server.
Set/get which WWW::RobotRules object to use.
Returns the number of documents fetched from this server host. Yes I know, this method should probably have been named num_visits() or something like that. :-(
Returns the number of seconds (from now) you must wait before you can make a new request to this host.
Returns a string that describes the state of the UA. Mainly useful for debugging.
LWP::UserAgent, WWW::RobotRules
Copyright 1996-2002 Gisle Aas.
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install LWP, copy and paste the appropriate command in to your terminal.
cpanm
cpanm LWP
CPAN shell
perl -MCPAN -e shell install LWP
For more information on module installation, please visit the detailed CPAN module installation guide.