WWW::Robot - configurable web traversal engine (for web robots & agents) River stage one • 2 direct dependents • 2 total dependents

This module implements a configurable web traversal engine, for a *robot* or other web agent. Given an initial web page (*URL*), the Robot will get the contents of that page, and extract all links on the page, adding them to a list of URLs to visit. ...

KVENTIN/WWW-Robot-0.026 - 07 Aug 2009 13:21:26 GMT

WWW::RobotRules - database of robots.txt-derived permissions River stage four • 4 direct dependents • 7663 total dependents

This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> Webmasters can use the /robots.txt file to forbid conforming robots from accessing parts of their web site. The pars...

GAAS/WWW-RobotRules-6.02 - 18 Feb 2012 13:09:13 GMT

WWW::SimpleRobot - a simple web robot for recursively following links on web pages. River stage zero No dependents

A simple perl module for doing robot stuff. For a more elaborate interface, see WWW::Robot. This version uses LWP::Simple to grab pages, and HTML::LinkExtor to extract the links from them. Only href attributes of anchor tags are extracted. Extracted ...

AWRIGLEY/WWW-SimpleRobot-0.07 - 28 Jun 2001 14:50:08 GMT

WWW::RobotRules::DBIC - Persistent RobotRules which use DBIC. River stage zero No dependents

WWW::RobotRules::DBIC is a subclass of WWW::RobotRules, which use DBIx::Class to store robots.txt info to any RDBMS....

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

WWW::RobotRules::Extended - database of robots.txt-derived permissions. This is a fork of WWW::RobotRules River stage zero No dependents

This module parses /robots.txt files as specified in "A Standard for Robot Exclusion", at <http://www.robotstxt.org/wc/norobots.html> It also parses rules that contains wildcards '*' and allow directives like Google does. Webmasters can use the /robo...

YSIMONX/WWW-RobotRules-Extended-0.02 - 14 Jan 2012 10:23:47 GMT

WWW::RobotRules::Parser - Just Parse robots.txt River stage zero No dependents

WWW::RobotRules::Parser allows you to simply parse robots.txt files as described in http://www.robotstxt.org/wc/norobots.html. Unlike WWW::RobotRules (which is very cool), this module does not take into consideration your user agent name when parsing...

DMAKI/WWW-RobotRules-Parser-0.04001 - 01 Dec 2007 13:33:54 GMT

WWW::RobotRules::Memcache - Use memcached in conjunction with WWW::RobotRules River stage zero No dependents

This is a subclass of WWW::RobotRules that uses Cache::Memcache to implement persistent caching of robots.txt and host visit information....

SOCK/WWW-RobotRules-Memcache-0.1 - 08 Sep 2006 02:02:39 GMT

WWW::Mixi - Perl extension for scraping the MIXI social networking service. River stage zero No dependents

WWW::Mixi uses LWP::RobotUA to scrape mixi.jp. This provide login method, get and put method, and some parsing method for user who create mixi spider. I think using WWW::Mixi is better than using LWP::UserAgent or LWP::Simple for accessing Mixi. WWW:...

TSUKAMOTO/WWW-Mixi-0.50 - 01 Aug 2007 06:02:56 GMT

lib/WWW/RobotRules/DBIC/Schema.pm River stage zero No dependents

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

WWW::OhNoRobotCom::Search - search comic transcriptions on http://ohnorobot.com River stage one • 1 direct dependent • 3 total dependents

The module provides interface to perform searches on <http://www.ohnorobot.com> comic transcriptions website....

ZOFFIX/WWW-OhNoRobotCom-Search-0.003 - 18 Dec 2013 22:59:56 GMT

WWW::RobotRules::AnyDBM_File - Persistent RobotRules River stage four • 4 direct dependents • 7663 total dependents

This is a subclass of *WWW::RobotRules* that uses the AnyDBM_File package to implement persistent diskcaching of robots.txt and host visit information. The constructor (the new() method) takes an extra argument specifying the name of the DBM file to ...

GAAS/WWW-RobotRules-6.02 - 18 Feb 2012 13:09:13 GMT

lib/WWW/RobotRules/DBIC/Schema/Rule.pm River stage zero No dependents

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

lib/WWW/RobotRules/DBIC/Schema/Netloc.pm River stage zero No dependents

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

lib/WWW/RobotRules/DBIC/Schema/DateTime.pm River stage zero No dependents

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

lib/WWW/RobotRules/DBIC/Schema/UserAgent.pm River stage zero No dependents

IKEBE/WWW-RobotRules-DBIC-0.01 - 18 Oct 2006 13:58:41 GMT

WWW::RobotRules::Parser::MultiValue - Parse robots.txt River stage zero No dependents

"WWW::RobotRules::Parser::MultiValue" is a parser for "robots.txt". Parsed rules for the specified user agent is stored as a Hash::MultiValue, where the key is a lower case rule name. "Request-rate" rule is handled specially. It is normalized to "Cra...

TARAO/WWW-RobotRules-Parser-MultiValue-0.02 - 12 Mar 2015 05:46:38 GMT

WWW::Scraper - framework for scraping results from search engines. River stage one • 1 direct dependent • 1 total dependent

NOTE: You can find a full description of the Scraper framework in WWW::Scraper::ScraperPOD.pm. "Scraper" is a framework for issuing queries to a search engine, and scraping the data from the resultant multi-page responses, and the associated detail p...

GLENNWOOD/Scraper-3.05 - 02 Aug 2003 07:47:12 GMT

POE::Component::WWW::OhNoRobotCom::Search - non-blocking POE based wrapper around WWW::OhNoRobotCom::Search module River stage one • 2 direct dependents • 2 total dependents

The module is a non-blocking wrapper around WWW::OhNoRobotCom::Search which provides interface to <http://www.ohnorobot.com/> search...

ZOFFIX/POE-Component-WWW-OhNoRobotCom-Search-0.002 - 17 Dec 2013 00:50:53 GMT

WWW::Link - maintain information about the state of links River stage zero No dependents

WWW::Link is a perl class which accepts and maintains information about links. For example, this would include urls which are referenced from a WWW page. The link class will be acted on by such programs as link checkers to give it information and by ...

MIKEDLR/WWW-Link-0.036 - 09 Feb 2002 18:10:43 GMT

WWW::Search - Virtual base class for WWW searches River stage two • 32 direct dependents • 34 total dependents

This class is the parent for all access methods supported by the "WWW::Search" library. This library implements a Perl API to web-based search engines. See README for a list of search engines currently supported, and for a lot of interesting high-lev...

MTHURN/WWW-Search-2.518 - 24 Nov 2018 12:03:31 GMT

40 results (0.036 seconds)