WWW::RobotRules - database of robots.txt-derived permissions River stage four • 4 direct dependents • 7667 total dependents

This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site. The pars...

GAAS/WWW-RobotRules-6.02 - 18 Feb 2012 13:09:13 GMT

WWW::RobotRules::DBIC - Persistent RobotRules which use DBIC. River stage zero No dependents

WWW::RobotRules::DBIC is a subclass of WWW::RobotRules, which use DBIx::Class to store robots.txt info to any RDBMS....

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

WWW::RobotRules::Extended - database of robots.txt-derived permissions. This is a fork of WWW::RobotRules River stage zero No dependents

This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> It also parses rules that contains wildcards '*' and allow directives like Google does. Webmasters can use the /robo...

YSIMONX/WWW-RobotRules-Extended-0.02 - 14 Jan 2012 10:23:47 GMT

WWW::RobotRules::Parser - Just Parse robots.txt River stage zero No dependents

WWW::RobotRules::Parser allows you to simply parse robots.txt files as described in http://www.robotstxt.org/wc/norobots.html. Unlike WWW::RobotRules (which is very cool), this module does not take into consideration your user agent name when parsing...

DMAKI/WWW-RobotRules-Parser-0.04001 - 01 Dec 2007 13:33:54 GMT

WWW::RobotRules::Memcache - Use memcached in conjunction with WWW::RobotRules River stage zero No dependents

This is a subclass of WWW::RobotRules that uses Cache::Memcache to implement persistent caching of robots.txt and host visit information....

SOCK/WWW-RobotRules-Memcache-0.1 - 08 Sep 2006 02:02:39 GMT

WWW::RobotRules::AnyDBM_File - Persistent RobotRules River stage four • 4 direct dependents • 7667 total dependents

This is a subclass of *WWW::RobotRules* that uses the AnyDBM_File package to implement persistent diskcaching of robots.txt and host visit information. The constructor (the new() method) takes an extra argument specifying the name of the DBM file to ...

GAAS/WWW-RobotRules-6.02 - 18 Feb 2012 13:09:13 GMT

WWW::RobotRules::Parser::MultiValue - Parse robots.txt River stage zero No dependents

"WWW::RobotRules::Parser::MultiValue" is a parser for "robots.txt". Parsed rules for the specified user agent is stored as a Hash::MultiValue, where the key is a lower case rule name. "Request-rate" rule is handled specially. It is normalized to "Cra...

TARAO/WWW-RobotRules-Parser-MultiValue-0.02 - 12 Mar 2015 05:46:38 GMT

WWW::Mixi - Perl extension for scraping the MIXI social networking service. River stage zero No dependents

WWW::Mixi uses LWP::RobotUA to scrape mixi.jp. This provide login method, get and put method, and some parsing method for user who create mixi spider. I think using WWW::Mixi is better than using LWP::UserAgent or LWP::Simple for accessing Mixi. WWW:...

TSUKAMOTO/WWW-Mixi-0.50 - 01 Aug 2007 06:02:56 GMT

lib/WWW/RobotRules/DBIC/Schema.pm River stage zero No dependents

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

lib/WWW/RobotRules/DBIC/Schema/Rule.pm River stage zero No dependents

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

lib/WWW/RobotRules/DBIC/Schema/Netloc.pm River stage zero No dependents

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

lib/WWW/RobotRules/DBIC/Schema/DateTime.pm River stage zero No dependents

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

lib/WWW/RobotRules/DBIC/Schema/UserAgent.pm River stage zero No dependents

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

WWW::Spyder - a simple non-persistent web crawler. River stage zero No dependents

ASHLEY/WWW-Spyder-0.24 - 27 Feb 2008 00:33:57 GMT

14 results (0.03 seconds)