Gungho - Yet Another High Performance Web Crawler Framework
use Gungho; Gungho->run($config);
Gungho is Yet Another Web Crawler Framework, aimed to be extensible and fast. Its meant to be a culmination of lessons learned while building Xango -- Xango was *fast*, but it was horribly hard to debug or to extend (Gungho even works right out of the box ;)
Therefore, Gungho's main aim is to make it as easy as possible to write complex crawlers, while still keeping crawling *fast*. You can simply specify the urls to fetch and some code to handle the responses -- we do the rest.
Gungho tries to build from clean structures, based upon principles from the likes of Catalyst and Plagger, so that you can easily extend it to your liking.
Features such as robot rules handling (robots.txt) and request throttling can be removed/added on the fly, just by specifying the components that you want to load. You can easily create additional functionality by writing your own component.
Gungho is still very fast -- it uses event driven frameworks such as POE, Danga::Socket, and IO::Async as the main engine to drive requests. Choose the best engine for your needs: For example, if you plan on creating a POE-based handler to process the response, you might choose the POE engine - it will fit nicely into the request cycle. However, do note that the most heavily excercised engine is POE. Danga::Socket and IO::Async works, but haven't been tested too vigorously. Please send in requests and bug reports if you encounter any problems.
WARNING: *ALL* APIs are still subject to change.
Gungho is comprised of three parts. A Provider, which provides Gungho with requests to process, a Handler, which handles the fetched page, and an Engine, which controls the entire process.
There are also "hooks". These hooks can be registered from anywhere by invoking the register_hook() method. They are run at particular points, which are specified when you call register_hook().
All components (engine, provider, handler) are overridable and switcheable. However, do note that if you plan on customizing stuff, you should be aware that Gungho uses Class::C3 extensively, and hence you may see warnings about the code you use.
--- debug: 1
Setting debug to a non-zero value will trigger debug messages to be displayed.
--- block_private_ip_address: 1
Setting this to a non-zero value will make addresses resolved via DNS lookups to be blocked, if they resolved to a private IP address such as 192.168.1.1. Note that 127.0.0.1 is also considered a private IP.
Components add new functionality to Gungho. Components are loaded at startup time fro the config file / hash given to Gungho constructor.
Gungho->run({ components => [ 'Throttle::Simple' ], throttle => { max_interval => ..., } });
Components modify Gungho's inheritance structure at run time to add extra functionality to Gungho, and therefore should only be loaded before starting the engine.
Here are some available components. Checkout the distribution for a current, complete list:
Handles collecting, parsing robots.txt, as well rejecting requests based on the rules provided from it.
Handles basic auth automatically.
Throttles requests based on the number of requests sent to a domain.
If you're looking into simple crawlers, you may want to look at Gungho::Inline,
Gungho::Inline->run({ provider => sub { ... }, handler => sub { ... } });
See the manual for Gungho::Inline for details.
Currently available hooks are:
This method has been deprecated. Use run() instead.
Starts the Gungho process. It requires either the name of a config filename or a hashref.
Returns true if Gungho supports some feature $name
Sets up the Gungho environment, including calling the various setup_* methods to configure the provider, engine, handler, etc.
Sets up the various components.
Registers a hook to be run under the specified $hook_name
Runs all the hooks under the hook $hook_name
Delegates to provider's has_requests
Delegates to provider's get_requests
Delegates to handler's handle_response
Calls provider->dispatch
Given a request, preps it before sending it to the engine
Delegates to engine's send_request
Loads the config from $config via Config::Any.
Loads a Gungho component. Compliments the module name with 'Gungho::$prefix::', unless the name is prefixed with a '+'. In that case, no transformation is performed, and the module name is used as-is.
One last note about Gungho - Don't use it if you are planning on accessing a single url -- It's usually not worth it, so you might as well use LWP::UserAgent or an equivalent module.
Gungho's event driven engine works best when you are accessing hundreds, if not thousands of urls. It may in fact be slower than using LWP::UserAgent if you are accessing just a single url.
Of course, you may wish to utilize features other than speed that Gungho provides, so at that point, it's simply up to you.
You can obtain the current code base from
http://gungho-crawler.googlecode.com/svn/trunk
Copyright (c) 2007 Daisuke Maki <daisuke@endeworks.jp>
Gungho::Inline Gungho::Component::RobotRules
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
See http://www.perl.com/perl/misc/Artistic.html
To install Gungho, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Gungho
CPAN shell
perl -MCPAN -e shell install Gungho
For more information on module installation, please visit the detailed CPAN module installation guide.