DynGig::Util::MapReduce - A Map Reduce Job Launcher
use DynGig::Util::MapReduce; my $batch = sub { .. }; my $map = sub { .. }; my $reduce = sub { .. }; my $job = DynGig::Util::MapReduce->new ( name => 'foo', batch => $batch, map => $map, reduce => $reduce, ); my %batch_param = ( .. ); my %map_param = ( .. ); my %reduce_param = ( .. ); my %context = ( .. ); $job->run ( context => \%context, batch => \%batch_param, map => \%map_param, reduce => \%reduce_param, ); my $result = $job->result();
Map/Reduce is an approach that collects data in parallel then processes data in serial. A Map/Reduce job has 4 components/steps. Each component is to be defined by the user.
Batch : divide the targets into batches. Map : create threads to collect data from/for each batch in parallel. Sort : aggregate collected data by status. Reduce: (optional) process aggregated data serially.
Returns the result of the run as a HASH reference or undef if invoked before run.
Returns the context of the run as a HASH reference or undef if invoked before run.
Returns the name of the job.
threads, Thread::Queue, and YAML::XS for data serialization.
See DynGig::Util
To install DynGig::Util, copy and paste the appropriate command in to your terminal.
cpanm
cpanm DynGig::Util
CPAN shell
perl -MCPAN -e shell install DynGig::Util
For more information on module installation, please visit the detailed CPAN module installation guide.