This script crawles a web page, and follows links with specified depth.
You can easily change
* a initial web page * the depth * how many crawlers
Moreover if you hack Crawler class, then it should be easy to implement
* whitelist, blacklist for links * priority for links
To install Parallel::Pipes, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Parallel::Pipes
CPAN shell
perl -MCPAN -e shell install Parallel::Pipes
For more information on module installation, please visit the detailed CPAN module installation guide.