Proc::Parallel::Unreliable - maintain a pool of unreliable slave processes
use Proc::Parallel::Unreliable; run_subprocess( input => $input_filehandle, output => $output_callback_function, cb => $slave_process_start_function, count => $number_of_slaves_to_run, timeout => $maximum_time_to_process_a_line, input_id => $callback_to_identify_input, input_hlines => $number_of_header_lines_in_input_stream, output_hlines => $number_of_header_lines_in_output_stream, output_id => $callback_to_identify_output, items => $number_of_buffer_lines_per_slave, track_items => $identify_items_flag, debug => $debug_level, pidfile => $process_id_file_to_update, );
This module deals with a particular situation: needing to run code that may crash or hang while processing a stream of data.
It reads input, farms the input out to one or more slave process, collects the output from the slave processes, and writes a single stream of output.
Each line of input must be self-contained because it can be randomly sent to any of the slaves.
It has two modes: tracking items, or not tracking items. When it tracks items, it uses the input_id callback to identify the input items and the output_id callback to identify the results it gets back from the slaves. When tracking items, the slave processes must produce output for each input item: they'll be killed off if they don't.
input_id
output_id
When not tracking items, output is sent to the slave processes until the buffers fill up. It's not as elegant.
If the input stream or the output stream has headers, the input stream headers will be reproduced for each slave and the output stream headers will be suppresssed except for one slave. Use the input_hlines and output_hlines parameters for this.
input_hlines
output_hlines
Output is written with a callbac: output.
output
The process id file, pidfile will be updated to show the current set of processes.
pidfile
run_subprocess( output => sub { print $output $o; }, input => $input, cb => sub { my ($my_input_fd, $my_output_fd) = @_; run_transformation($my_input_fd,$my_output_fd); }, count => 3, timeout => 60, input_id => sub { my ($iref) = @_; $$iref =~ /^(\d+)/ return $1; }, input_hlines => 0, output_hlines => 1, output_id => sub { my ($oref) = @_; $$iref =~ /^(\d+)/ return $1; }, items => 4, track_items => 1, debug => $debug, pidfile => "/var/run/myprocess.pid", );
This package may be used and redistributed under the terms of either the Artistic 2.0 or LGPL 2.1 license.
To install Proc::Parallel, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Proc::Parallel
CPAN shell
perl -MCPAN -e shell install Proc::Parallel
For more information on module installation, please visit the detailed CPAN module installation guide.