Gungho::Component::RobotRules - Respect robots.txt
--- components: - RobotRules
Returns 1 if request is allowed to be fetched, 0 if not. -1 and -2 are returned when there is a pending request to fetch /robots.txt
Dispatches requests that were pending because of a missing robots.txt entry
Pushes a request in the wait queue for a robots.txt
Parses the robot rule and stores it
To install Gungho, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Gungho
CPAN shell
perl -MCPAN -e shell install Gungho
For more information on module installation, please visit the detailed CPAN module installation guide.