Gearman::Driver - Manages Gearman workers
package My::Workers::One; # Yes, you need to do it exactly this way use base qw(Gearman::Driver::Worker); use Moose; # this method will be registered with gearmand as 'My::Workers::One::scale_image' sub scale_image : Job { my ( $self, $job, $workload ) = @_; # do something } # this method will be registered with gearmand as 'My::Workers::One::do_something_else' sub do_something_else : Job : MinProcesses(2) : MaxProcesses(15) { my ( $self, $job, $workload ) = @_; # do something } # this method wont be registered with gearmand at all sub do_something_internal { my ( $self, $job, $workload ) = @_; # do something } 1; package My::Workers::Two; use base qw(Gearman::Driver::Worker); use Moose; # this method will be registered with gearmand as 'My::Workers::Two::scale_image' sub scale_image : Job { my ( $self, $job, $workload ) = @_; # do something } 1; package main; use Gearman::Driver; my $driver = Gearman::Driver->new( namespaces => [qw(My::Workers)], server => 'localhost:4730,otherhost:4731', interval => 60, ); $driver->run;
Having hundreds of Gearman workers running in separate processes can consume a lot of RAM. Often many of these workers share the same code/objects, like the database layer using DBIx::Class for example. This is where Gearman::Driver comes in handy:
You write some base class which inherits from Gearman::Driver::Worker. Your base class loads your database layer for example. Each of your worker classes inherit from that base class. In the worker classes you can register single methods as jobs with gearmand. It's even possible to control how many workers doing that job/method in parallel. And this is the point where you'll save some RAM: Instead of starting each worker in a separate process Gearman::Driver will fork each worker from the main process. This will take advantage of copy-on-write on Linux and save some RAM.
There's only one mandatory parameter which has to be set when calling the constructor: namespaces
use Gearman::Driver; my $driver = Gearman::Driver->new( namespaces => [qw(My::Workers)] );
See also: namespaces. If you do not set server (gearmand) attribute the default will be used: localhost:4730
localhost:4730
Each module found in your namespaces will be loaded and introspected, looking for methods having the 'Job' attribute set:
package My::Workers::ONE; sub scale_image : Job { my ( $self, $job, $workload ) = @_; # do something }
This method will be registered as job function with gearmand, verify it by doing:
plu@mbp ~$ telnet localhost 4730 Trying ::1... Connected to localhost. Escape character is '^]'. status My::Workers::ONE::scale_image 0 0 1 . ^] telnet> Connection closed.
If you dont like to use the full package name you can also specify a custom prefix:
package My::Workers::ONE; sub prefix { 'foo_bar_' } sub scale_image : Job { my ( $self, $job, $workload ) = @_; # do something }
This would register 'foo_bar_scale_image' with gearmand.
See also: prefix
Will be passed to Module::Find findallmod method to load worker modules. Each one of those modules has to be inherited from Gearman::Driver::Worker or a subclass of it. It's also possible to use the full package name to load a single module/file. There is also a method get_namespaces which returns a sorted list of all namespaces.
findallmod
See also: "wanted".
isa: ArrayRef
ArrayRef
required: True
True
isa: CodeRef
CodeRef
required: False
False
This CodeRef will be called on each of the modules found in your "namespace". The first and only parameter to this sub is the name of the module. If a true value is returned, the module will be loaded and checked if it's a valid Gearman::Driver::Worker subclass.
Let's say you have a namespace called My::Project:
My::Project
My::Project::Web
My::Project::Web::Controller::Root
My::Project::Web::Controller::Admin
My::Project::Web::Controller::User
My::Project::Web::Model::DBIC
My::Project::Worker::ScaleImage
My::Project::Worker::RemoveUser
To avoid every module being loaded and inspected being a Gearman::Driver::Worker subclass you can use wanted to only load classes having Worker in the package name:
wanted
Worker
my $driver = Gearman::Driver->new( interval => 0, namespaces => [qw(My::Project)], wanted => sub { return 1 if /Worker/; return 0; }, );
This would only load:
A list of Gearman servers the workers should connect to. The format for the server list is: host[:port][,host[:port]]
host[:port][,host[:port]]
See also: Gearman::XS
default: localhost:4730
isa: Str
Str
Gearman::Driver has a telnet management console, see also:
Gearman::Driver::Console
default: 47300
47300
isa: Int
Int
Set this to 0 to disable management console at all.
0
Each n seconds Net::Telnet::Gearman is used in Gearman::Driver::Observer to check status of free/running/busy workers on gearmand. This is used to fork more workers depending on the queue size and the MinProcesses/MaxProcesses attribute of the job method. See also: Gearman::Driver::Worker
default: 5
5
Path to logfile.
default: gearman_driver.log
gearman_driver.log
See also Log::Log4perl.
default: [%d] %p %m%n
[%d] %p %m%n
default: INFO
INFO
This is just for convenience to extend @INC from command line using gearman_driver.pl:
@INC
gearman_driver.pl
gearman_driver.pl --lib ./lib --lib /custom/lib --namespaces My::Workers
Whenever Gearman::Driver::Observer sees a job that isnt handled it will call this CodeRef, passing following arguments:
$driver
$status
my $driver = Gearman::Driver->new( namespaces => [qw(My::Workers)], unknown_job_callback => sub { my ( $driver, $status ) = @_; # notify nagios here for example } );
$status might look like:
$VAR1 = { 'busy' => 0, 'free' => 0, 'name' => 'GDExamples::Sleeper::unknown_job', 'queue' => 6, 'running' => 0 };
This might be interesting for subclassing Gearman::Driver.
Every worker module loaded by Module::Find will be added to this list. There are also two methods: get_modules and has_modules.
readonly: True
Stores all Gearman::Driver::Job instances. The key is the name the job gets registered with gearmand. There are also two methods: get_job and has_job.
Example:
{ 'My::Workers::ONE::scale_image' => bless( {...}, 'Gearman::Driver::Job' ), 'My::Workers::ONE::do_something_else' => bless( {...}, 'Gearman::Driver::Job' ), 'My::Workers::TWO::scale_image' => bless( {...}, 'Gearman::Driver::Job' ), }
isa: HashRef
HashRef
Instance of Gearman::Driver::Observer.
isa: Gearman::Driver::Observer
Gearman::Driver::Observer
Instance of Gearman::Driver::Console.
isa: Gearman::Driver::Console
There's one mandatory param (hashref) with following keys:
decode (optionally)
Name of a decoder method in your worker object.
encode (optionally)
Name of a encoder method in your worker object.
method (mandatory)
Reference to a Class::MOP::Method object which will get invoked.
min_processes (mandatory)
Minimum number of processes that should be forked.
max_processes (mandatory)
Maximum number of processes that may be forked.
name (mandatory)
Job name/alias that method should be registered with Gearman.
object (mandatory)
Object that should be passed as first parameter to the job method.
Basically you never really need this method if you use "namespaces". But "namespaces" depend on method attributes which some people do hate. In this case, feel free to setup your $driver this way:
package My::Workers::One; use Moose; use JSON::XS; extends 'Gearman::Driver::Worker::Base'; sub scale_image { my ( $self, $job, $workload ) = @_; # do something } # this method will be registered with gearmand as 'My::Workers::One::do_something_else' sub do_something_else { my ( $self, $job, $workload ) = @_; # do something } sub encode_json { my ( $self, $result ) = @_; return JSON::XS::encode_json($result); } sub decode_json { my ( $self, $workload ) = @_; return JSON::XS::decode_json($workload); } 1; package main; use Gearman::Driver; use My::Workers::One; my $driver = Gearman::Driver->new( server => 'localhost:4730,otherhost:4731', interval => 60, ); my $worker = My::Workers::One->new(); foreach my $method (qw(scale_image do_something_else)) { $driver->add_job( decode => 'decode_json', encode => 'encode_json', max_processes => 5, method => $worker->meta->find_method_by_name($method)->body, min_processes => 1, name => $method, object => $worker, ); } $driver->run;
This must be called after the Gearman::Driver object is instantiated.
Sends TERM signal to all child processes and exits Gearman::Driver.
Returns a sorted list of namespaces.
Returns a sorted list of modules.
Returns the count of modules.
Params: $name
Returns true/false if the job exists.
Returns the job instance.
Johannes Plunien <plu@cpan.org>
Uwe Voelker, <uwe.voelker@gmx.de>
Copyright 2009 by Johannes Plunien
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
Gearman::Driver::Job
Gearman::Driver::Worker
Gearman::XS
Log::Log4perl
Module::Find
Moose
MooseX::Getopt
MooseX::Log::Log4perl
MooseX::MethodAttributes
Net::Telnet::Gearman
POE
http://www.gearman.org/
http://github.com/plu/gearman-driver/
To install Gearman::Driver, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Gearman::Driver
CPAN shell
perl -MCPAN -e shell install Gearman::Driver
For more information on module installation, please visit the detailed CPAN module installation guide.