MultiProcFactory - Base class for multiprocess batch processing.
#!/usr/bin/perl use strict; use MultiProcFactory; my $do_child = sub { my $self = shift; $self->inc_scalar(); $self->set_hash_element($self->get_prockey() => " $$: " . $self->get_scalar()); }; my $do_parent_final = sub { my $self = shift; foreach my $key ($self->get_prockeys()) { my $value = $self->get_hash_element($key); $self->log_parent("$key: $value\n"); } }; my $link_obj = MultiProcFactory->factory( work_by => 'MultiProcFactory::Generic', do_child => $do_child, do_parent_final => $do_parent_final, partition_list => [ 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H' ] ); $link_obj->run();
This is a factory framework interface for multiprocess batch processing. You need to write a class to inherit from this class that fits your data model. Can be a very powerful data processing tool, for system wide application patterns.
This class is a factory class for multiprocess batch processing. The definition of processing bins are defined in subclasses that this object returns. run method manages child processes and executes code references. Depending on subclass logic can be used to execute do_child as an iterator for batch processing. Shared memory through IPC::Shareable (SysV IPC) is available by default. I have setup two shared variables, a scalar and a hash.
This method takes all contructor arguments. Additional parameters will be needed for your subclassed object.
work_by =>'BFI::MultiProcFactory::Schema::Mailing' ## Package name to subclass
do_child => $code_ref_a ## code executed in each child
do_parent_final => $code_ref_b, ## code executed by parent when child procs are complete.
max_proc => N # max number of concurrent child processes (default 20)
log_children => 0|1 (default 0)
log_parent => 0|1 (default 1)
do_parent_init => $code_ref_c, ## code executed in parent before forking
parent_sig => {INT => $coderef, TERM => $coderef, ...}
child_sig => {INT => $coderef, TERM => $coderef, ...}
IPC_OFF => 0|1 (default 0) ## turns off default allocation of shared memory
This method is called after initialization. It contains all forking and subroutine execution logic.
Method logs input string to parent filehandle
Method logs input string to child filehandle
Default - $0 minus any extensions . '.log' can override default by redefining in subclass.
Default - $0 minus any extensions . "_$instance\.log" can override default by redefining in subclass.
returns current childs process key. This key maps back to process slot in partition_hash. Has no meaning if called from parent and should return undef.
returns list of process keys in partition_hash used for iterating over all children
wrapper for IPC::Shareable shlock() on shared scalar
wrapper for IPC::Shareable shlock() on shared hash
wrapper for IPC::Shareable shunlock() on shared scalar
wrapper for IPC::Shareable shunlock() on shared hash
wrapper to set shared hash with key => value. Calls hash_lock() and hash_unlock()
wrapper to get value stored in shared hash identified by $key
wrapper to set shared scalar var with $value
wrapper to increment current value in shared scalar by 1
wrapper to decrement value in shared scalar by 1
wrapper to access shared scalar value
called internally by factory()
sets child code reference
sets parent initialization reference
sets parent cleanup code reference
sets parent signal handlers if passed in with hash ref parent_sig =>{}, this allows you to override the default signal handling behavior.
sets child signal handlers if passed in with hash ref parent_sig =>{}, this allows you to override the default signal handling behavior.
* Parent - by default TERM, ABRT, INT and QUIT are set to call IPC::Shareable->clean_up. Unless you like calling ipcrm this is a good thing.
* Child - by default TERM, ABRT, INT and QUIT are reset undef.
* Sets up two shared variables with IPC::Shareable, a scalar and a hash.
* For the curious semaphores and memory are stored in
$self->{share_scalar}{handle}
$self->{share_scalar}{var}
$self->{share_hash}{handle}
$self->{share_hash}{var}
$self->{prockey} - defines each process bin
called from constructor. parent contains partitioning algorithm. Partition algorithm bins data into self->{partition_hash} Each of these bins is forked.
This method does any basic child process level initialization.
This method at the bare minimum must call do_child(). Can be written to iterate do_child over a result set.
Aaron Dancygier, <adancygier@bigfootinteractive.com>
Copyright 2004 by Aaron Dancygier
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
perl(1), IPC::Shareable, MultiProcFactory::Generic
To install MultiProcFactory, copy and paste the appropriate command in to your terminal.
cpanm
cpanm MultiProcFactory
CPAN shell
perl -MCPAN -e shell install MultiProcFactory
For more information on module installation, please visit the detailed CPAN module installation guide.