The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.
 / 
CrawlerCommons-RobotRulesParser-0.03
River stage zero No dependents
++

Changes for version 0.03

  • A version to hopefully get propagated to metacpan as well

Modules

CrawlerCommons::RobotRules
the result of a parsed robots.txt
CrawlerCommons::RobotRulesParser
parser for robots.txt files

Provides

CrawlerCommons::ParseState
in lib/CrawlerCommons/ParseState.pm
CrawlerCommons::RobotDirective
in lib/CrawlerCommons/RobotDirective.pm
CrawlerCommons::RobotRule
in lib/CrawlerCommons/RobotRules.pm
CrawlerCommons::RobotToken
in lib/CrawlerCommons/RobotToken.pm

Other files