NAME

MCE::Shared - MCE extension for sharing data between workers

VERSION

This document describes MCE::Shared version 1.699_008

SYNOPSIS

   # OO construction

   use MCE::Shared Sereal => 1;

   my $ar = MCE::Shared->array( @list );
   my $cv = MCE::Shared->condvar( 0 );
   my $fh = MCE::Shared->handle( '>>', \*STDOUT );
   my $ha = MCE::Shared->hash( @pairs );
   my $db = MCE::Shared->minidb();
   my $oh = MCE::Shared->ordhash( @pairs );
   my $qu = MCE::Shared->queue( await => 1, fast => 0 );
   my $va = MCE::Shared->scalar( $value );
   my $nu = MCE::Shared->sequence( $begin, $end, $step, $fmt );
   my $ob = MCE::Shared->share( $blessed_object );

   # Tie construction

   use MCE::Flow;
   use MCE::Shared Sereal => 1;
   use feature 'say';

   tie my $var, 'MCE::Shared', 'initial value';
   tie my @ary, 'MCE::Shared', qw(a list of values);
   tie my %has, 'MCE::Shared', (key1 => 'value', key2 => 'value');

   tie my $cnt, 'MCE::Shared', 0;
   tie my @foo, 'MCE::Shared';
   tie my %bar, 'MCE::Shared';

   my $m1 = MCE::Mutex->new;

   mce_flow {
      max_workers => 4
   },
   sub {
      my ($mce) = @_;
      my ($pid, $wid) = (MCE->pid, MCE->wid);

      ## Locking is required when multiple workers update the same element.
      ## This requires 2 trips to the manager process (fetch and store).

      $m1->synchronize( sub {
         $cnt += 1;
      });

      ## Locking is not necessary when updating unique elements.

      $foo[ $wid - 1 ] = $pid;
      $bar{ $pid }     = $wid;

      return;
   };

   say "scalar : $cnt";
   say " array : $_" for (@foo);
   say "  hash : $_ => $bar{$_}" for (sort keys %bar);

   -- Output

   scalar : 4
    array : 37847
    array : 37848
    array : 37849
    array : 37850
     hash : 37847 => 1
     hash : 37848 => 2
     hash : 37849 => 3
     hash : 37850 => 4

DESCRIPTION

This module provides data sharing for MCE supporting threads and processes.

MCE::Shared enables extra functionality on systems with IO::FDPass. Without it, MCE::Shared is unable to send file descriptors to the shared-manager process for queue, condvar, and possibly handle.

As of this writing, the IO::FDPass module is not a requirement for running MCE::Shared nor is the check made during installation. The reason is that IO::FDPass is not possible on Cygwin and not sure about AIX.

The following is a suggestion for systems without IO::FDPass. This restriction applies to queue, condvar, and handle only.

   use MCE::Shared;

   # Construct shared queue(s) and condvar(s) first.
   # These contain GLOB handles - freezing not allowed.

   my $q1  = MCE::Shared->queue();
   my $q2  = MCE::Shared->queue();

   my $cv1 = MCE::Shared->condvar();
   my $cv2 = MCE::Shared->condvar();

   # Start the shared-manager manually.

   MCE::Shared->start();

   # The shared-manager process knows of STDOUT, STDERR, STDIN

   my $fh1 = MCE::Shared->handle(">>", \*STDOUT);  # ok
   my $fh2 = MCE::Shared->handle("<", "/path/to/sequence.fasta");  # ok
   my $h1  = MCE::Shared->hash();

Otherwise, sharing is immediate and not delayed with IO::FDPass. It is not necessary to share queue and condvar first or worry about starting the shared-manager process.

   use MCE::Shared;

   my $h1 = MCE::Shared->hash();    # shares immediately
   my $q1 = MCE::Shared->queue();   # IO::FDPass sends file descriptors
   my $cv = MCE::Shared->condvar(); # IO::FDPass sends file descriptors
   my $h2 = MCE::Shared->ordhash();

array

condvar

handle

hash

minidb

ordhash

queue

scalar

sequence

array, condvar, handle, hash, minidb, ordhash, queue, scalar, and sequence are sugar syntax for constructing a shared object.

  # long form

  use MCE::Shared;
  use MCE::Shared::Array;
  use MCE::Shared::Hash;

  my $ar = MCE::Shared->share( MCE::Shared::Array->new() );
  my $ha = MCE::Shared->share( MCE::Shared::Hash->new() );

  # short form

  use MCE::Shared;

  my $ar = MCE::Shared->array( @list );
  my $cv = MCE::Shared->condvar( 0 );
  my $fh = MCE::Shared->handle( '>>', \*STDOUT );
  my $ha = MCE::Shared->hash( @pairs );
  my $db = MCE::Shared->minidb();
  my $oh = MCE::Shared->ordhash( @pairs );
  my $qu = MCE::Shared->queue( await => 1, fast => 0 );
  my $va = MCE::Shared->scalar( $value );
  my $nu = MCE::Shared->sequence( $begin, $end, $step, $fmt );

num_sequence

num_sequence is an alias for sequence.

share

This class method transfers the blessed-object to the shared-manager process and returns a MCE::Shared::Object containing the SHARED_ID. The object must not contain any GLOB's or CODE_REF's or the transfer will fail.

Unlike threads::shared, objects are not deeply shared. The shared object is accessable only through the underlying OO interface.

   use MCE::Shared;
   use Hash::Ordered;

   my ($ho_shared, $ho_unshared);

   $ho_shared = MCE::Shared->share( Hash::Ordered->new() );

   $ho_shared->push( @pairs );            # OO interface only
   $ho_shared->mset( @pairs );

   $ho_unshared = $ho_shared->export();   # back to unshared
   $ho_unshared = $ho_shared->destroy();  # including destruction

The following provide long and short forms for constructing a shared array, hash, or scalar object.

   use MCE::Shared;

   use MCE::Shared::Array;    # Loading helper classes is not necessary
   use MCE::Shared::Hash;     # when using the shorter form.
   use MCE::Shared::Scalar;

   my $a1 = MCE::Shared->share( MCE::Shared::Array->new( @list ) );
   my $a3 = MCE::Shared->share( [ @list ] );  # sugar syntax
   my $a2 = MCE::Shared->array( @list );

   my $h1 = MCE::Shared->share( MCE::Shared::Hash->new( @pairs ) );
   my $h3 = MCE::Shared->share( { @pairs } ); # sugar syntax
   my $h2 = MCE::Shared->hash( @pairs );

   my $s1 = MCE::Shared->share( MCE::Shared::Scalar->new( 20 ) );
   my $s2 = MCE::Shared->share( \do{ my $o = 20 } );
   my $s4 = MCE::Shared->scalar( 20 );

pdl_byte

pdl_short

pdl_ushort

pdl_long

pdl_longlong

pdl_float

pdl_double

pdl_ones

pdl_sequence

pdl_zeroes

pdl_indx

pdl

pdl_byte, pdl_short, pdl_ushort, pdl_long, pdl_longlong, pdl_float, pdl_double, pdl_ones, pdl_sequence, pdl_zeroes, pdl_indx, and pdl are sugar syntax for PDL construction take place under the shared-manager process.

   use PDL;
   use PDL::IO::Storable;   # must load for freezing/thawing

   use MCE::Shared;         # must load MCE::Shared after PDL
   
   # not efficient from memory copy/transfer and unnecessary destruction
   my $ob1 = MCE::Shared->share( zeroes( 256, 256 ) );

   # efficient
   my $ob1 = MCE::Shared->zeroes( 256, 256 );

ins_inplace

The ins_inplace method applies to shared PDL objects. It supports two forms for writing bits back into the PDL object residing under the shared-manager process.

   # --- action taken by the shared-manager process
   # ins_inplace(  2 args ):   $this->slice( $arg1 ) .= $arg2;
   # ins_inplace( >2 args ):   ins( inplace( $this ), $what, @coords );

   # --- use case
   $o->ins_inplace( ":,$start:$stop", $result );  #  2 args
   $o->ins_inplace( $result, 0, $seq_n );         # >2 args

The MCE-Cookbook on Github provides a couple working PDL demonstrations for further reading.

https://github.com/marioroy/mce-cookbook

COMMON API

blessed

Returns the real blessed name, provided by the shared-manager process.

   use Scalar::Util qw(blessed);
   use MCE::Shared;

   use MCE::Shared::Ordhash;
   use Hash::Ordered;

   my $oh1 = MCE::Shared->share( MCE::Shared::Ordhash->new() );
   my $oh2 = MCE::Shared->share( Hash::Ordered->new() );

   print blessed($oh1), "\n";    # MCE::Shared::Object
   print blessed($oh2), "\n";    # MCE::Shared::Object

   print $oh1->blessed(), "\n";  # MCE::Shared::Ordhash
   print $oh2->blessed(), "\n";  # Hash::Ordered

destroy

Exports optionally, but destroys the shared object entirely from the shared-manager process.

   my $exported_ob = $shared_ob->destroy();

   $shared_ob; # becomes undef

export

export ( keys )

Exports the shared object into a non-shared object. One must export when passing the shared object into any dump routine. Otherwise, the data ${ SHARED_ID } is all one will see.

   use MCE::Shared;
   use MCE::Shared::Ordhash;

   sub _dump {
      require Data::Dumper unless $INC{'Data/Dumper.pm'};
      no warnings 'once';

      local $Data::Dumper::Varname  = 'VAR';
      local $Data::Dumper::Deepcopy = 1;
      local $Data::Dumper::Indent   = 1;
      local $Data::Dumper::Purity   = 1;
      local $Data::Dumper::Sortkeys = 0;
      local $Data::Dumper::Terse    = 0;

      print Data::Dumper::Dumper($_[0]) . "\n";
   }

   # these do the same thing
   my $oh1 = MCE::Shared->share( MCE::Shared::Ordhash->new() );
   my $oh2 = MCE::Shared->ordhash();

   _dump($oh1);  # ${ 1 }  # SHARED_ID value
   _dump($oh2);  # ${ 2 }

   _dump($oh1->export());  # actual structure and content
   _dump($oh2->export());

export can optionally take a list of indices/keys for what to export. This applies to shared array, hash, and ordhash.

   use MCE::Shared;

   my $h1 = MCE::Shared->hash(           # shared hash
      qw/ I Heard The Bluebirds Sing by Marty Robbins /
        # k v     k   v         k    v  k     v
   );

   my $h2 = $h1->export( qw/ I The / );  # non-shared hash

   _dump($h2);

   __END__

   $VAR1 = bless( {
     'I' => 'Heard',
     'The' => 'Bluebirds'
   }, 'MCE::Shared::Hash' );

next

rewind

rewind ( "query string" )

rewind ( begin, end, [ step, format ] )

next and rewind enable parallel iteration between workers for shared array, hash, ordhash, and sequence. Call rewind after running to reset the pointer.

   use MCE::Hobo;
   use MCE::Shared;

   my $ob = MCE::Shared->array( 'a' .. 'j' );

   sub parallel {
      my ($id) = @_;
      while (defined (my $item = $ob->next)) {
         print "$id: $item\n";
         sleep 1;
      }
   }

   MCE::Hobo->new( \&parallel, $_ ) for 1 .. 3;

   # ... do other work ...

   $_->join() for MCE::Hobo->list();

   -- Output

   1: a
   2: b
   3: c
   2: f
   1: d
   3: e
   2: g
   3: i
   1: h
   2: j

There are two forms for iterating through a shared hash or ordhash object. The next method is wantarray-aware providing key and value in list context and value only in scalar context.

   use MCE::Hobo;
   use MCE::Shared;

   my $ob = MCE::Shared->ordhash(
      map {( "key_$_" => "val_$_" )} "a" .. "j"
   );

   sub iter1 {
      my ($id) = @_;
      while ( my ($key, $val) = $ob->next ) {
         print "$id: $key => $val\n";
         sleep 1;
      }
   }

   sub iter2 {
      my ($id) = @_;
      while ( defined (my $val = $ob->next) ) {
         print "$id: $val\n";
         sleep 1;
      }
   }

   MCE::Hobo->new(\&iter1, $_) for 1 .. 3;
   $_->join() for MCE::Hobo->list();

   $ob->rewind();

   MCE::Hobo->new(\&iter2, $_) for 1 .. 3;
   $_->join() for MCE::Hobo->list();

Although the shared-manager process iterates orderly, there is no guarantee for the amount of time required by workers. Thus, output may not be ordered.

   -- Output

   1: key_a => val_a
   2: key_b => val_b
   3: key_c => val_c
   1: key_d => val_d
   3: key_f => val_f
   2: key_e => val_e
   1: key_g => val_g
   3: key_i => val_i
   2: key_h => val_h
   1: key_j => val_j
   1: val_a
   2: val_b
   3: val_c
   3: val_f
   1: val_d
   2: val_e
   3: val_h
   1: val_g
   2: val_i
   3: val_j

store ( key, value )

Deep-sharing non-blessed structure(s) is possible with store only. store, an alias to STORE, converts non-blessed deeply-structures to shared objects recursively.

   use MCE::Shared;

   my $h1 = MCE::Shared->hash();
   my $h2 = MCE::Shared->hash();

   # auto-shares deeply
   $h1->store( 'key', [ 0, 2, 5, { 'foo' => 'bar' } ] );
   $h2->{key}[3]{foo} = 'baz';   # via auto-vivification

   my $v1 = $h1->get('key')->get(3)->get('foo');  # bar
   my $v2 = $h2->get('key')->get(3)->get('foo');  # baz
   my $v3 = $h2->{key}[3]{foo};                   # baz

Each level in a deeply structure requires a separate trip to the shared-manager processs. There is a faster way if the app calls for just HoH and/or HoA. The included MCE::Shared::Minidb module provides optimized methods for working with HoH and HoA structures.

See MCE::Shared::Minidb.

SERVER API

start

Starts the shared-manager process. This is done automatically.

   MCE::Shared->start();

stop

Stops the shared-manager process wiping all shared data content. This is not typically done by the user, but rather by END automatically when the script terminates.

   MCE::Shared->stop();

init

This is called automatically by each MCE/Hobo worker immediately after being spawned. The effect is extra parallelism during inter-process communication. The optional ID (an integer) is modded in a round-robin fashion.

   MCE::Shared->init( ID );
   MCE::Shared->init();

INDEX

MCE, MCE::Core

AUTHOR

Mario E. Roy, <marioeroy AT gmail DOT com>

To install MCE, copy and paste the appropriate command in to your terminal.

cpanm

cpanm MCE

CPAN shell

perl -MCPAN -e shell
install MCE

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

VERSION

SYNOPSIS

DESCRIPTION

DATA SHARING

OBJECT SHARING

PDL SHARING

COMMON API

SERVER API

INDEX

AUTHOR

Module Install Instructions