The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
Gungho-0.08013
River stage one • 1 direct dependent • 1 total dependent
++

Changes for version 0.08013

  • RobotRules
    • Add expiration configuration parameters so that ttl for each robot rules can be configured
    • Add get_pending_robots_txt() and push_pending_robots_txt(). Pending requests are now controled in the Storage::* classes
    • Fix calling API for Storage::Cache, so that it also works for Cache::Memcached::Managed
    • Fix a problem where RobotRules::Cache (which is distributed) couldn't figure out that a robots.txt request has been dispatched.

Documentation

gungho
An Extensible, High-Performance Web Crawler Framework

Modules

Gungho
Yet Another High Performance Web Crawler Framework
Gungho::Base
Base Class For Various Gungho Objects
Gungho::Component
Component Base Class For Gungho
Gungho::Component::Authentication
Base Class For WWW Authentication
Gungho::Component::Authentication::Basic
Add Basic Auth To Gungho
Gungho::Component::Cache
Use Cache In Your App
Gungho::Component::RobotRules
Respect robots.txt
Gungho::Component::RobotRules::Rule
A Rule Object
Gungho::Component::RobotRules::Storage
RobotRules Storage Base Class
Gungho::Component::RobotRules::Storage::Cache
Cache Storage For RobotRules
Gungho::Component::RobotRules::Storage::DB_File
DB_File Storage For RobotRules
Gungho::Component::Throttle
Base Class To Throttle Requests
Gungho::Component::Throttle::Domain
Throttle By Domain
Gungho::Component::Throttle::Simple
Throttle By Number Of Requests
Gungho::Component::Throttle::Throttler
Data::Throttler Based Throttling
Gungho::Engine
Base Class For Gungho Engine
Gungho::Engine::Danga::Socket
Gungho Engine Using Danga::Socket
Gungho::Engine::IO::Async
IO::Async Engine
Gungho::Engine::POE
POE Engine For Gungho
Gungho::Exception
Gungho Exceptions
Gungho::Handler
Base Class For Gungho Handlers
Gungho::Handler::FileWriter::Simple
Write Out Fetched Contents To File
Gungho::Handler::Null
A Handler That Does Nothing
Gungho::Inline
Inline Your Providers And Handlers
Gungho::Log
Log Base Class For Gungho
Gungho::Log::Dispatch
Log::Dispatch-Based Log For Gungho
Gungho::Log::Simple
Simple Gungho Log Class
Gungho::Plugin
Gungho Plugin Base Class
Gungho::Plugin::RequestLog
Log Requests
Gungho::Plugin::RequestTimer
Keep Track Of Time To Finish Request
Gungho::Provider
Base Class For Gungho Prividers
Gungho::Provider::File::Simple
Provide Requests From A Simple File
Gungho::Provider::Simple
An In-Memory, Simple Provider
Gungho::Provider::YAML
Specify requests in YAML format
Gungho::Request
A Gungho Request Object
Gungho::Request::http
HTTP specific utilities

Provides

Gungho::Engine::IO::Async::Impl::Select
in lib/Gungho/Engine/IO/Async.pm
Gungho::Handler::Inline
in lib/Gungho/Inline.pm
Gungho::Provider::Inline
in lib/Gungho/Inline.pm

Examples

Other files