WWW::Crawler::Mojo::Queue::Memory - Crawler queue with memory
Crawler queue with memory.
This class inherits all methods from WWW::Crawler::Mojo::Queue and implements following new ones.
An subroutine reference called on enqueue process for avoiding redundant requests. It marks the job 'done' and returns 0, and next time returns 1.
if (!$queue->redundancy->($job)) { $queue->enqueue($job); }
Defaults to a code that uses "no cleanup" storage. By replacing this you can control the memory usage.
$queue->redundancy(sub { my $d = $_[0]->digest; return 1 if $your_storage{$d}; $your_storage{$d} = 1; return; });
jobs.
This class inherits all methods from WWW::Crawler::Mojo::Queue class and implements following new ones.
Implementation for WWW::Crawler::Mojo::Queue interface.
Keita Sugama, <sugama@jamadam.com>
Copyright (C) Keita Sugama.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install WWW::Crawler::Mojo, copy and paste the appropriate command in to your terminal.
cpanm
cpanm WWW::Crawler::Mojo
CPAN shell
perl -MCPAN -e shell install WWW::Crawler::Mojo
For more information on module installation, please visit the detailed CPAN module installation guide.