The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
App-Prove-Plugin-Distributed-0.08
River stage zero No dependents
++
/ App::Prove::Plugin::Distributed

NAME

App::Prove::Plugin::Distributed - an App::Prove plugin to distribute test jobs using client and server model.

VERSION

Version 0.08

SYNOPSIS

  # All of the examples below is loading tests into the worker perl processes
  # If you want to run the tests in a separate perl process, you can specify
  # the '--detach' option to accomplish that.
  
  # Default workers with L<IPC::Open3> as worker processes.
  prove -PDistributed -j2 t/

  # Distributed jobs with LSF workers.
  prove -PDistributed --distributed-type=LSF -j2 t/

  # Distributed jobs with SSH workers.
  prove -PDistributed --distributed-type=SSH -j2 --hosts=host1,host2 t/

  # If you are using home network that does not have name server setup,
  # you can specify the option --use-local-public-ip
  prove -PDistributed --distributed-type=SSH --use-local-public-ip -j2 --hosts=host1,host2 t/
  
  # Distributed jobs with PBS workers using L<PBS::Client>. Note: This is not tested yet.
  prove -PDistributed --distributed-type=PBS -j2 t/

  # Distributed jobs with PBS workers using L<PBS::Client>. Note: This is not tested yet.
  # With PBS option
  prove -PDistributed --distributed-type=PBS --mem=200 -j2 t/

DESCRIPTION

A plugin for App::Prove to distribute job. The core implementation of the plugin is to provide a easy interface and functionality to extend the use of any distribution method.

The initiate release of this module was using the idea from FCGI::Daemon that load perl code file using "do" perl function to the worker perl process to execute tests.

Currently, the distribution comes with a few implementation of distribution methods to initiate external "worker" processes. Shown below is the list.

   L<IPC::Open3>
   LSF 
   SSH
   L<PBS::Client>  * Note: PBS implemetation is not tested yet.

FUNCTIONS

Basic functions.

load

Load the plugin configuration. It will setup all of the tests to be distributed through the TAP::Parser::SourceHandler::Worker source handler class.

extra_used_libs

Return a list of paths in @INC that are not part of the compiled-in lsit of paths

start_server

Start a server to serve the test.

Parameter is the contoller peer address.

trigger_end_blocks_before_child_process_exit

Trigger END blocks before the child process exit. The main reason is to have the Test::Builder to have change to finish up.

rsync_test_env

Rsync test enviroment to the worker host.

Parameters $app object Returns boolean

Plugin Command Line Options

manager

Option only for the worker process to indicate the control process that started the worder. The format is hostname:port

Example, --manager=myhostname:1234

distributed-type

Specify the distribution method. Currently, the valid values are shown below.

   LSF
   PBS
   SSH

start-up

Start up script to load to the worker process before running test. It is only without the --detach option.

Example,

    --start-up=/dir/path/setup_my_enviroment_variable.pl

tear-down

Currently, the tear-down option will not be run.

error-log

Capture any error from the worker. The error log file is the file path that the worker can write to.

detach

Detach the executing of test from the worker process. Currently, the test job will be executed with exec perl function.

use-local-public-ip

If you are using home network that does not have Domain Name System (DNS) or name server setup, you can specify the option use-local-public-ip to find out the local public ip address of your machine that you start the prove. This option is boolean option and does not take argument.

sync-type

To distribute your project/program files to the worker machine, you can use the sync-type option. Currently, it only supports rsync type.

Example,

   prove -PDistributed --distributed-type=SSH --hosts="192.168.1.100,192.168.1.101"\
    --sync-type=rsync --use-local-public-ip -I lib -j2 --detach --recurse

source-dir

Source project/program files that you want to distribute to the worker machine. If it is not specified, the current directory is assumed.

   prove -PDistributed --distributed-type=SSH --hosts="192.168.1.100,192.168.1.101" \
   --sync-type=rsync --use-local-public-ip --source-dir=/my/home/project \
   -I lib -j2 --detach --recurse

destination-dir

Destination directory on the worker machine that you want the project/program files to distribute to. If it is not specified, the system will use File::Temp::tempdir to create a directory to be distributed to.

Example,

   prove -PDistributed --distributed-type=SSH --hosts="192.168.1.100,192.168.1.101" \
   --sync-type=rsync --use-local-public-ip --destination-dir=/my/worker/home/project \
   -I lib -j2 --detach --recurse

Worker Specific Options

Each worker can have its specific options. Please refer to the particular source handler worker module for detail.

BUGS

Currently, the only known bug is when running in the worker perl process without using detach option with the empty string regular expression match. Shown below is the example code that will generate the bug. For more information please check out the test t/sample-tests/empty_string_problem.

    ok('good=bad' =~ m/^.*?=.*/, 'test');

    ok('test' =~ m//, 'this will failed before the previous regex match ' . 
                      'with a "?=" regex match. I have no way to reset ' .
                      'back the previous regex change the regex engine ' .
                      'unless I put it in its scope.');

CAVEAT

If the plugin is used with a shared DBI handle and without the detach option, the database handle will be closed by the child process when the child process exits.

Shown below is an example,

   prove -PDistributed -j2 --start-up=InitShareDBI.pm t/

Shown below is the InitShareDBI.pm codes.

   package InitShareDBI;

   use strict;
   use warnings;
   use DBI;
   
   unless($InitShareDBI::dbh) {
       $InitShareDBI::dbh = DBI->connect('dbi:mysql:test;mysql_socket=/tmp/mysql.sock', 'user', 'password', {PrintError => 1});
   }
   
   sub dbh {
       return $InitShareDBI::dbh;
   }
   
   1;

To get around the problem of database handle close. Shown below is one of the solution.

    $InitShareDBI::dbh->{InactiveDestroy} = 1;

or,

    {
        no warnings 'redefine';
        *DBI::db::DESTROY = sub {

            # NO OP.
        };
    }

The complete codes for the InitShareDBI.pm file below.

    package InitShareDBI;

    use strict;
    use warnings;
    use DBI;

    #LSF: This has to be loaded when loading this module.
    unless ($InitShareDBI::dbh) {
        $InitShareDBI::dbh =
          DBI->connect( 'dbi:mysql:database=test;host=127.0.0.1;port=3306',
            'test', 'test', { PrintError => 1 } );
        #LSF: This will prevent it to be destroy.
        #$InitShareDBI::dbh->{InactiveDestroy} = 1;
    }

    {
        no warnings 'redefine';
        *DBI::db::DESTROY = sub {

            # NO OP.
        };
    }

    sub dbh {
        return $InitShareDBI::dbh;
    }

    1;

Please refer to "DBI, fork, and clone" <http://www.perlmonks.org/?node_id=594175> for more information on this.

AUTHORS

Shin Leong <lsf@cpan.org>

CREDITS

Many thanks to Anthony Brummett who has contributed ideas, code, and helped out on the module since it's initial release. See the github <https://github.com/shin82008/App-Prove-Plugin-Distributed> for details.

LICENCE AND COPYRIGHT

Copyright (c) 2012 Shin Leong <lsf@cpan.org>. All rights reserved.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.12.4 or, at your option, any later version of Perl 5 you may have available. See perlartistic.