WWW::Crawler::Mojo - A web crawling framework for Perl
use strict; use warnings; use WWW::Crawler::Mojo; my $bot = WWW::Crawler::Mojo->new; $bot->on(res => sub { my ($bot, $scrape, $job, $res) = @_; $bot->enqueue($_) for $scrape->('#context'); }); $bot->enqueue('http://example.com/'); $bot->crawl;
WWW::Crawler::Mojo is a web crawling framework for those who are familiar with Mojo::* APIs.
Althogh the module is only well tested for "focused crawl" at this point, you can also use it for endless crawling by taking special care of memory usage.
WWW::Crawler::Mojo inherits all attributes from Mojo::EventEmitter and implements the following new ones.
A number of main event loop interval in milliseconds. Defaults to 0.25.
$bot->clock_speed(2); my $clock = $bot->clock_speed; # 2
An amount of max connections.
$bot->max_conn(5); say $bot->max_conn; # 5
An amount of max connections per host.
$bot->max_conn_per_host(5); say $bot->max_conn_per_host; # 5
WWW::Crawler::Mojo::Queue::Memory object for default.
$bot->queue(WWW::Crawler::Mojo::Queue::Memory->new); $bot->queue->enqueue($job);
An interval in seconds to shuffle the job queue. It also evalutated as boolean for disabling/enabling the feature. Defaults to undef, meaning disable.
$bot->shuffle(5); say $bot->shuffle; # 5
A WWW::Crawler::Mojo::UserAgent instance.
my $ua = $bot->ua; $bot->ua(WWW::Crawler::Mojo::UserAgent->new);
Name of crawler for User-Agent header.
$bot->ua_name('my-bot/0.01 (+https://example.com/)'); say $bot->ua_name; # 'my-bot/0.01 (+https://example.com/)'
WWW::Crawler::Mojo inherits all events from Mojo::EventEmitter and implements the following new ones.
Emitted when crawler got response from server. The callback takes 4 arguments.
$bot->on(res => sub { my ($bot, $scrape, $job, $res) = @_; if (...) { $bot->enqueue($_) for $scrape->(); } else { # DO NOTHING } });
WWW::Crawler::Mojo instance.
Scraper code reference for current document. The code takes optional argument CSS selector for context and returns new jobs.
for my $job ($scrape->($context)) { $bot->enqueue($job) }
Optionally you can specify a scraping target container in CSS selector.
@jobs = $scrape->('#container'); @jobs = $scrape->(['#container1', '#container2']);
WWW::Crawler::Mojo::Job instance.
Mojo::Message::Response instance.
Emitted when queue length gets zero.
$bot->on(empty => sub { my ($bot) = @_; say "Queue is drained out."; });
Emitted when user agent returns no status code for request. Possibly caused by network errors or un-responsible servers.
$bot->on(error => sub { my ($bot, $error, $job) = @_; say "error: $_[1]"; if (...) { # until failur occures 3 times $bot->requeue($job); } });
Note that server errors such as 404 or 500 cannot be catched with the event. Consider res event for the use case instead of this.
Emitted right before crawl is started.
$bot->on(start => sub { my $self = shift; ... });
WWW::Crawler::Mojo inherits all methods from Mojo::EventEmitter and implements the following new ones.
Starts crawling loop.
$bot->crawl;
Initializes crawler settings.
$bot->init;
Processes a job.
$bot->process_job;
Displays starting messages to STDOUT
$bot->say_start;
Parses and discovers links in a web page and CSS. This performs scraping. With the optional 4th argument, you can specify a CSS selector to container you would collect URLs within.
$bot->scrape($res, $job, ); $bot->scrape($res, $job, $selector); $bot->scrape($res, $job, [$selector1, $selector2]);
Stop crawling.
$bot->stop;
Appends one or more URLs or WWW::Crawler::Mojo::Job objects.
$bot->enqueue('http://example.com/index1.html');
OR
$bot->enqueue($job1, $job2);
$bot->enqueue( 'http://example.com/index1.html', 'http://example.com/index2.html', 'http://example.com/index3.html', );
Appends one or more URLs or jobs for re-try. This accepts same arguments as enqueue method.
$self->on(error => sub { my ($self, $msg, $job) = @_; if (...) { # until failur occures 3 times $bot->requeue($job); } });
Collects URLs out of HTML.
$bot->collect_urls_html($dom, sub { my ($uri, $dom) = @_; });
https://github.com/jamadam/WWW-Flatten
Keita Sugama, <sugama@jamadam.com>
Copyright (C) jamadam
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install WWW::Crawler::Mojo, copy and paste the appropriate command in to your terminal.
cpanm
cpanm WWW::Crawler::Mojo
CPAN shell
perl -MCPAN -e shell install WWW::Crawler::Mojo
For more information on module installation, please visit the detailed CPAN module installation guide.