The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
Parallel-ForkManager-Segmented-0.0.1
River stage one • 2 direct dependents • 3 total dependents
2 ++
/ Parallel::ForkManager::Segmented

NAME

Parallel::ForkManager::Segmented - use Parallel::ForkManager on batches / segments of items.

VERSION

version 0.0.1

SYNOPSIS

    #! /usr/bin/env perl

    use strict;
    use warnings;
    use 5.014;
    use Cwd ();

    use WML_Frontends::Wml::Runner ();
    use Parallel::ForkManager::Segmented ();

    my $UNCOND  = $ENV{UNCOND} // '';
    my $CMD     = shift @ARGV;
    my (@dests) = @ARGV;

    my $PWD       = Cwd::getcwd();
    my @WML_FLAGS = (
        qq%
    --passoption=2,-X3074 --passoption=2,-I../lib/ --passoption=3,-I../lib/ --passoption=3,-w -I../lib/ $ENV{LATEMP_WML_FLAGS} -p1-3,5,7 -DROOT~. -DLATEMP_THEME=sf.org1 -I $HOME/apps/wml
    % =~ /(\S+)/g
    );

    my $T2_SRC_DIR = 't2';
    my $T2_DEST    = "dest/$T2_SRC_DIR";

    chdir($T2_SRC_DIR);

    my $obj = WML_Frontends::Wml::Runner->new;

    sub is_newer
    {
        my $file1 = shift;
        my $file2 = shift;
        my @stat1 = stat($file1);
        my @stat2 = stat($file2);
        if ( !@stat2 )
        {
            return 1;
        }
        return ( $stat1[9] >= $stat2[9] );
    }

    my @queue;
    foreach my $lfn (@dests)
    {
        my $dest     = "$T2_DEST/$lfn";
        my $abs_dest = "$PWD/$dest";
        my $src      = "$lfn.wml";
        if ( $UNCOND or is_newer( $src, $abs_dest ) )
        {
            push @queue, [ [ $abs_dest, "-DLATEMP_FILENAME=$lfn", $src, ], $dest ];
        }
    }
    my $to_proc = [ map $_->[1], @queue ];
    my @FLAGS = ( @WML_FLAGS, '-o', );
    my $proc = sub {
        $obj->run_with_ARGV(
            {
                ARGV => [ @FLAGS, @{ shift(@_)->[0] } ],
            }
        ) and die "$!";
        return;
    };
    Parallel::ForkManager::Segmented->new->run(
        {
            WITH_PM      => 1,
            items        => \@queue,
            nproc        => 4,
            batch_size   => 8,
            process_item => $proc,
        }
    );
    system("cd $PWD && $CMD @{$to_proc}") and die "$!";

VERSION

version 0.0.1

METHODS

my $obj = Parallel::ForkManager::Segmented->new;

Initializes a new object.

$obj->run(+{ %ARGS });

Runs the processing. Accepts the following named arguments:

  • process_item

    A reference to a subroutine that accepts one item and processes it.

  • items

    A reference to the array of items.

  • nproc

    The number of child processes to use.

  • batch_size

    The number of items in each batch.

  • disable_fork

    Disable forking and use of Parallel::ForkManager and process the items serially.

AUTHOR

Shlomi Fish <shlomif@cpan.org>

COPYRIGHT AND LICENSE

This software is Copyright (c) 2018 by Shlomi Fish.

This is free software, licensed under:

  The MIT (X11) License

BUGS

Please report any bugs or feature requests on the bugtracker website https://github.com/shlomif/parallel-forkmanager-segmented/issues

When submitting a bug or request, please include a test-file or a patch to an existing test-file that illustrates the bug or desired feature.

SUPPORT

Perldoc

You can find documentation for this module with the perldoc command.

  perldoc Parallel::ForkManager::Segmented

Websites

The following websites have more information about this module, and may be of help to you. As always, in addition to those websites please use your favorite search engine to discover more resources.

Bugs / Feature Requests

Please report any bugs or feature requests by email to bug-parallel-forkmanager-segmented at rt.cpan.org, or through the web interface at https://rt.cpan.org/Public/Bug/Report.html?Queue=Parallel-ForkManager-Segmented. You will be automatically notified of any progress on the request by the system.

Source Code

The code is open to the world, and available for you to hack on. Please feel free to browse it and play with it, or whatever. If you want to contribute patches, please send me a diff or prod me to pull from your repository :)

https://github.com/shlomif/perl-Parallel-ForkManager-Segmented

  git clone https://github.com/shlomif/perl-Parallel-ForkManager-Segmented.git