Revision history for MCE

1.505 Thu Jan 21 01:30:00 EST 2014


  * Delete $self->{input_data} inside the worker immediately during spawning
    (if input_data is an ARRAY, GLOB, or Iterator reference).

  * Reverted the logic for the RS (record separator) option from the 1.4x
    code base. This is now working as expected.

  * Specifying chunk_size => 'auto' via the 'init' method (all 5 models)
    causes MCE validation to croak due to 'auto' being carried over to the
    core API. Note that only the 5 models are allowed 'auto' for chunk_size
    as of this time.

  * Update to the MCE->do method. Previously, an undef sent back from the
    callback function ended up as a blank value "". Both "" and undef are
    now properly captured and sent back to the worker process.

  * Added an if statement inside the abort method.
  * Removed an old unlink _store.db statement (old code left behind).
  * Removed an unused _next variable inside


  * Enhanced to handle additional options including recursion.
    This script now supports many egrep options [ceHhiLlmnqRrsv].

  * IO performance for examples/ was improved. The real focus here is
    demonstrating output order.

  * The chunk_size option can take a suffix; K (Kilobytes) or M (Megabytes).

  * The following examples except --max-workers=NUM --chunk-size=NUM options.,,,, and


  * The input_data option can now receive an iterator reference. Added a
    new example for demonstraton. In addition, there are
    several examples listed under a new section "SYNTAX for INPUT_DATA"
    under MCE::Core.pod.

  * Added a new demo script; bin/mce_grep. This is a wrapper script for the
    grep binary. This script supports agrep, grep, egrep, fgrep, & tre-agrep.
    Simply create a link to mce_grep or make a copy. Both Windows and Cygwin
    are supported as well. Recursion works for all binaries including agrep
    (-R, -r options).

      ln mce_grep mce_agrep
      ln mce_grep mce_egrep
      ln mce_grep mce_fgrep
      ln mce_grep mce_tre-agrep

    Try with the --lang=C option for faster execution time (-i runs faster).
    Try mce_agrep or mce_tre-agrep against very large files. The speedup is
    linear and makes good utilization of all available cores on the box.

    One may specify the chunking level via the --chunk-level option. For large
    files, specify 'file' (chunks file). For many small files, use 'list'.

    I have tested against the following GnuWin32 packages found at this URL.         ## Contains agrep.exe (also runs under Cygwin)        ## Contains egrep.exe, fgrep.exe, grep.exe  ## These are required for the GnuWin32 binaries.

    Btw, bin/mce_grep is optional and therefore not installed by default
    when running make install for the MCE module. Simply copy mce_grep as for Windows. The .pl is optional for other environments.

1.504 Tue Oct 29 16:00:00 EST 2013


  * MCE::Grep, MCE::Map, and MCE::Stream were failing for mce_grep_s,
    mce_map_s, and mce_stream_s when specifying chunk_size => 1.


  * Calibrated 'auto' slightly in MCE::Util::_parse_chunk_size.

1.503 Mon Oct 28 17:00:00 EST 2013


  * The previous release introduced a bug by removing the line closing
    STDERR and STDOUT prior to workers exiting. The proper fix intended
    is to flush (not close) the handles. Closing them was not the thing
    to do in the first place due to possibly needed by the END block or
    the WARN/DIE handlers initiated from inside the END block.


  * Updated README under examples/tbray and examples/matmult.
  * Changed 0.499 to 0.5 inside the yield method.
  * Small cosmetic changes otherwise.

1.502 Mon Oct 21 16:00:00 EST 2013


  * The END block for all models will return immediately when called by a
    worker thread or process. Removed the line closing STDERR and STDOUT
    prior to workers exiting. It turns out that workers were calling
    shutdown which is not allowed. This impacted MCE::Flow, MCE::Grep,
    MCE::Loop, MCE::Map, and MCE::Stream.

  * An update was applied to DESTROY in MCE::Queue to address an edge
    case during additional testing across several environments.

  * Added documentation describing the core methods in MCE. I had moved the
    missing sections to another file some time back and totally forgotten
    about them when finalizing on MCE::Core.pod for the 1.5 release.


  * All models will croak when specifying an invalid MCE option.


  * One can specify a hash reference for the gather option. Updated
    documentation describing the gather option and the gather method.

1.501 Wed Oct 16 01:20:00 EST 2013


  * Addressed an issue with the worker signal "die" handler (#89538). Eval'd
    code should not raise an exception causing the app to die. This is working
    as expected for workers spawned as children.

    For threads, a die called explicitly inside an eval block will cause the
    worker to exit (same as previously). However, an eval 'use MissingModule'
    will no longer cause the thread to die.


  * Changelog section uses a new format beginning with this release.


  * Added examples/ for demonstrating MCE::Flow, MCE::Queue, and

1.500 Thu Oct 10 01:00:00 EST 2013

  * MCE 1.5 is backwards compatible with 1.4 and below.

  * The documentation, previously MCE.pod, moved to MCE::Core.pod with the
    examples section placed in MCE::Examples.pod. The MCE.pod file serves
    as an index page for the various documentation.

  * IPC has been enhanced with 8 data channels. Many operations run 3x when
    compared with MCE 1.4.

  * Five models: MCE::Flow, MCE::Grep, MCE::Loop, MCE::Map, and MCE::Stream.

  * Hybrid queues via MCE::Queue allowing for normal and priority queues.

  * MCE::Subs for exporting functions prefixed with mce_; e.g. mce_wid.

  * All public methods can be called directly using the package name
    and method e.g. MCE->wid, MCE->run.

          max_workers => 'auto',
          user_func => sub {
             my $wid = MCE->wid;
             MCE->sendto("STDOUT", "Hello from $wid\n");


  * Localize the input scalar $_ prior to calling user_func. Folks can
    use $_ for input_data and sequence of numbers. Added section to docs
    explaining DEFAULT INPUT SCALAR.

  * New options (bounds_only, gather, interval, task_name).

    The task_end option can now be specified at the top level.
    See docs for use case.

    Input_data can be specified inside the first task instead of having
    to specify this at the top level. Input_data is ignored when specified
    for tasks other than the first task.

       user_tasks => [{
          input_data => \@list,

  * New public methods
    chunk_id, gather, freeze, thaw, yield, task_name, print, printf, say

  * New example (

  * Optimized the and examples. These run much faster.
    Try these out against large log files. Both examples fly.

  * Barrier synchronization update. Two sockets are utilized instead of
    2 lock files. This, now works wonderfully under the Cygwin environment.
    For threading, the removal of 2 lock files increases the number of
    threads allowed from about 1/3rd previously to under 1/2 of ulimit,
    e.g. int(ulimit -n / 2 - 20).

  * Removed the logic for determining MAX_OPEN_FILES and MAX_USER_PROCS.
    MCE no longer has a constraint on max_workers allowed.

  * Code re-factor work. Added a private method _validate_runstate called
    by various methods. Organized the code slightly such as placement of

1.415 Mon Jun 17 15:00:00 EST 2013

  * Code-refactor in preparation for the upcoming 1.5 release.
    This completes the 1.4x branch for a very solid and stable release.

  * IPC optimization. Localize $\ and $/ only as needed. Small tweak to
    logic when specifying RS (record separator) for input data.

    Foreach is very communication intensive. Forseq is less so. Chunking
    was made faster by increasing chunk size from 500 to 2500. I've reached
    my goal for forseq by reaching 60,000+. A while back, foreach could
    barely reach 18,000.

       Before and after results with IPC optimization.

                             (Before)      (After)
       Parallel::Loops          600          600
       $mce->foreach         20,000       21,500  (+  1,500)
       $mce->forseq          55,000       64,000  (+  9,000)
       $mce->forchunk       395,000      450,000  (+ 55,000)

  * Updated the process method. Allows sequence to be specified as an
    option. Also allow the hash to be specified as the 1st or 2nd

       $mce->process( \@input_array, { options } );
       $mce->process( { options }, [ 1..1000 ] );
       $mce->process( { sequence => [ 20, 40, 2 ] } );

  * The forseq method will now honor chunk_size greater than 1. Updated
    docs with use case.

  * Added CONST short for EXPORT_CONST, e.g. use MCE CONST => 1;

1.414 Sun Jun 16 13:00:00 EST 2013

  * Upstream bug fixes. Mainly chunk_size was not honored when specifying
    user_tasks and sequence/chunk_size for the first task. Bug is with
    not setting abort_msg correctly.

1.413 Mon Jun 10 02:00:00 EST 2013

  * Fixed typo: Changed local @_ to local $@ inside _parse_max_workers
    before evaling.

1.412 Sun Jun 09 14:00:00 EST 2013

  * Updated the main README file and CHANGES under 1.411 below.

  * Added support for multiplication and division as well as
    case insentivity to the private _parse_max_workers method.

       max_workers => 'auto',      ## = MCE::Util::get_ncpu()
       max_workers => 'Auto-1',    ## = MCE::Util::get_ncpu() - 1
       max_workers => 'AUTO + 3',  ## = MCE::Util::get_ncpu() + 3
       max_workers => 'AUTO * 1.5',
       max_workers => 'auto / 1.333',

1.411 Sun Jun 09 02:00:00 EST 2013

  * Reverted the change made in MCE-1.410. That degraded slurp IO on
    large files. This restores slurp IO performance in MCE.

  * Corrected an if statement towards the end of the shutdown method.
    The delay was occurring always when it was met only if MCE was
    launched from inside a non-main thread.

  * Addressed a race condition with barrier synchronization under the
    Cygwin environment. Was not able to reproduce this with other OS's.
    This takes barrier synchronization work to 100%. Delay statements
    for Cygwin were removed around barrier synchronization.

  * Optimized writes to sockets. Arguments passed to print statements
    are concatenated as one big string. Removed local $/ = $LF when
    reading from the queue socket. It's not required there since using
    read and known size. The example (very communication
    intensive) can now do 20000 in 1 second on my Macbook Pro. Updated
    benchmark results inside,, and

  * MAX_WORKERS, CHUNK_SIZE, TMP_DIR, FREEZE and THAW can be specified
    when loading the module. FREEZE and THAW allows one to choose an
    alternative serialization module if preferred for your project.

       use Sereal qw(encode_sereal decode_sereal);
       use MCE FREEZE => \&encode_sereal, THAW => \&decode_sereal;

  * Passing EXPORT_CONST => 1 will export 3 constants: SELF, CHUNK, CID
    The "my ($self, $chunk_ref, $chunk_id) = @_" line is not necessary.
    $_[SELF], $_[CHUNK], $_[CID] refers to $_[0], $_[1], $_[2].

  * The MCE::Util module was created. It contains the get_ncpu function.
    This is largely borrowed from MCE supports
    'auto' when specifying max_workers. Read doc for other use case.

       max_workers => 'auto';  ## Same as MCE::Util::get_ncpu()

  * Re-factored code in preparation for the upcoming 1.5 release. The
    1.5 release will use just 3 socket pairs versus 4. I did not want
    to include that change in this release. Therefore, I decided to
    take the upstream bug fixes and apply them to the 1.4 base.

    Re-factored documentation by making use of the full 78 character
    width per line. In addition tidbits on the new features mentioned

  * Updated the main README file under the top level dir.

1.410 Tue May 28 23:30:00 EST 2013

  * Use threads under MSWin32 for 02* and 03* test scripts.
  * Removed sysopen/sysseek/sysread to simplify logic due to negligible
    performance gains over open/seek/read.
  * Minor updates to documentation.

1.409 Sun May 12 22:45:00 EST 2013

  * Croak if user_func is not defined and input_data/sequence is specified.
  * Fix barrier synchronization when running multiple tasks via user_tasks.
  * Updated Perl documentation for easier reading through cpan/metapan.
  * Renamed private method _worker_sequence to _worker_sequence_queue.

1.408 Tue Mar 19 22:00:00 EST 2013

  * Minor tweaks here and there to further increase reliability.
  * Updated the barrier synchronization logic to not stall. Updated the
    perl docs on mixing "sync" with "do" or "sendto" methods.
  * Added new "status" method for the manager process.
  * Added new arguments for MCE::Signal: -no_kill9, -no_sigmsg

1.407 Thu Mar 14 21:00:00 EST 2013

  * This marks a tremendous effort in achieving parity across the board
    from Cygwin to Windows and obviously UNIX. MCE now works beautifully
    under the Cygwin environment including Perl for Windows.
  * Up'ed the maximum workers allowed for both Cygwin and Windows to
    56 (forking) and 80 (threading).
  * Barrier synchronization requires an extra semaphore file, therefore
    the maximum workers allowed under the UNIX environment for threading
    decreased. It shouldn't be a problem as there are many threads already.
  * Addressed an issue with barrier synchronization under Cygwin.
  * Addressed an issue with the die handler for the main worker method
    when threading.
  * Thank you for hanging in there. It took quite some time to get there.
    This is the most stable release thus far for MCE across the board.

1.406 Tue Mar 12 19:00:00 EST 2013

  * Added support for barrier synchronization (via new sync method).
    Added example.
  * Addressed rounding errors with the sequence generator.
    The sequence option now follows a bank-teller queuing model when
    generating numbers. This applies to task 0 only. Other tasks can
    still specify sequence where numbers will be distributed equally
    among workers like before.
  * Optimized the _worker_request_chunk private method.
  * A take 2 on the matrix multiplication examples. This is better
    organized with updated README file to include the script running
    time in the results.

1.405 Mon Mar 04 19:00:00 EST 2013

  * Added in the event folks cannot make use of /dev/shm
    used by the example.
  * Optimized the send method -- workers process immediately after receiving
    data. Updated run times in README for the strassen examples.
  * MCE no longer calls setpgrp by default as of MCE 1.405. There is only
    one reason to call setpgrp, but many reasons not to. The sole reason
    was for MCE to run correctly with Daemon::Control. If needed, one can
    pass the option to MCE::Signal qw(-setpgrp).
  * Return void in the shutdown method (previously was returning $self).
  * Tidy code inside sequence generator.

1.404 Sun Feb 24 13:00:00 EST 2013

  * Added sess_dir method
  * Completed work with matmult/* examples
    Added, removed
    Added strassen_pdl_o/p/q/r/ 
    Added benchmark results from a 32-way box at the end of the readme
  * Removed lines setting max limit for files/procs

1.403 Sun Feb 17 15:00:00 EST 2013

  * Wrap sub PDL::CLONE_SKIP into a no warnings 'redefine' block
    MCE now works with PDL::Parallel::threads without any warnings
  * Added missing examples/matmult/ to MANIFEST
  * Refactored strassen examples, memory consumption reduced by > than 50%
  * Added -- uses PDL::Parallel::threads to share matrices
  * Added -- matrix b is read from shared memory, not mmap
  * Added -- additional improvements to memory reduction
  * Added -- shown running with 4 workers (half and half)
  * Re-ran matrix multiplication examples and updated results in README file
  * Added -no_setpgrp option to
    Ctrl-C does not respond when running /usr/bin/time
  * Added undef $buffer in a couple of places within
  * Added David Mertens and Adam Sj√łgren to CREDITS
  * The send method now checks if sending > total workers after spawning
    not before

1.402 Thr Feb 14 07:30:00 EST 2013

  * Updated matrix multiplication examples including README
  * Added examples/matmult/

1.401 Tue Feb 12 19:00:00 EST 2013

  * Added sub PDL::CLONE_SKIP { 1 } to Running PDL + MCE threads no
    longer crashes during exiting.
  * Updated matrix multiplication examples. All examples now work under the
    windows environment no matter if threading or forking. Unix is stable as
    well if wanting to use PDL + MCE and use_threads => 1 or 0.
  * Added benchmark results for 2048x2048, 4096x4096, and 8192x8192 to the
    README file under examples/matmult/
  * Updated documentation

1.400 Mon Feb 11 07:00:00 EST 2013

  * Slight optimization in the _do_callback method
  * Added 2 new options: user_args and RS (record separator)
  * Added new send method for sending data to workers after spawning and
    prior to running
  * The sequence option can now take an ARRAY reference
  * Updated documentation on new features
  * Added matrix multiplication examples

1.306 Sat Jan 05 18:00:00 EST 2013

  * Added if statement around setpgrp(0,0). That function is not supported
    under Windows.
  * Updated logic for removing any remaining MCE session directories inside

1.305 Sat Jan 05 16:00:00 EST 2013

  * Added check for $^S to the DIE handler inside the _worker_main method
  * Added setpgrp(0,0) to MCE::Signal's BEGIN block
  * MCE::Signal points to a _mce_sess_dir hash in the event of a signal,
    will remove the sess_dir(s) as well. This is needed when tmp_dir is
    specified during instantiation and pointing to another location than

1.304 Wed Jan 02 22:00:00 EST 2013

  * Added Oliver Gorwits to CREDITS for identifying 2 issues
  * Direct die to CORE::die inside handler if executing an eval
  * Undef $mce_spawned_ref if signal was caught (stop_and_exit)
  * Changed INIT to sub import in

1.303 Tue Jan 01 20:00:00 EST 2013

  * Bump version -- VERSION now matches with META.yml
  * Sorted forchunk, foreach, forseq methods inside
  * Modified if statement in run method
  * Task ID is never undef, therefore removed check inside
    restart_worker and worker_do methods
  * Added 2 package variables to MCE::Signal
    $display_die_with_localtime and $display_warn_with_localtime
  * Completed updates to documentation
  * Update to and examples

1.302 Tue Jan 01 07:30:00 EST 2013

  * Fixed validation logic for sequence
  * Updated the sequence generator -- now supports chunking
  * Updated example to demo user_tasks, sequence, and
    chunk_size applied together
  * Documentation updates

1.301 Mon Dec 31 15:00:00 EST 2012

  * Emphasis on documentation -- better flow plus additional clarity
  * Minor update to sequence option validation
  * Minor update to included examples

1.300 Mon Dec 31 06:00:00 EST 2012

  * New methods...: chunk_size, restart_worker, task_id, task_wid, tmp_dir
  * New options...: on_post_exit, on_post_run, sequence
  * New examples..:,
  * Overhaul to exit method
    Workers can exit or die without impacting the manager process
  * Enabled executable bit for test files
  * Removed localtime output in die and warn handlers
  * All 3 delay options are consistent whether or not user_tasks is specified
  * Removed logic around total_ended count -- replaced with new exit logic
  * Code refactoring plus documentation updates
  * Added LICENSE file

1.201 Fri Dec 21 00:00:00 EST 2012

  * Added MCE.pod -- moved documentation from to pod file
  * Added missing use strict/warnings to test scripts
  * Default to 1 for chunk_size and max_workers if not specified
  * Test::More is not a requirement to run MCE, only for building
  * Changed the format for the change log file

1.200 Thu Dec 20 00:00:00 EST 2012

  * Added new user_tasks option
  * Added space between method name and left-paren for header lines in POD
  * Remove not-needed BSD::Resource and forks inside BEGIN/INIT blocks

1.106 Wed Dec 19 05:00:00 EST 2012

  * Added t/pod-coverage.t
  * Big overhaul of the MCE documentation -- all methods are documented
  * Croak if method suited for a MCE worker is called by the main MCE process
  * Croak if method suited for the main MCE process is called by a MCE worker
  * Updated Makefile.PL to declare the minimum Perl version

1.105 Sun Dec 16 23:00:00 EST 2012

  * Completed code re-factoring
  * Added t/pod.t

1.104 Sun Nov 25 17:00:00 EST 2012

  * Added 1 new example to MCE's Perl documentation
  * Use module::method name versus constant symbol when calling _croak
  * Croak if session directory is not writeable inside MCE::spawn
  * Renamed _mce_id to _mce_sid (met to be spawn id actually)
  * Re-calibrated maximum workers allowed

1.103 Fri Nov 23 13:00:00 EST 2012

  * Added writeable check on /dev/shm
  * Croak if tmp dir is not writeable inside MCE::Signal::import

1.102 Thu Nov 22 13:00:00 EST 2012

  * Woohoot !!! MCE now passes with Perl 5.17.x
  * Added Copying file -- same as in Perl

1.101 Wed Nov 21 16:00:00 EST 2012

  * Shifted white space to the left for code blocks inside documentation

1.100 Wed Nov 21 10:00:00 EST 2012

  * Completed optimization and shakeout for MCE's existing API
  * File handles are cached when calling sendto and appending to a file
  * The sendto method now supports multiple arguments -- see perldoc
  * Added new option: flush_file

1.008 Sat Nov 17 23:00:00 EST 2012

  * Update on __DIE__ and __WARN__ handling in MCE. This addresses the 
    unreferenced scalars seen in packaging logs at for
    Perl under Windows:
  * Update t/01_load_signal_arg.t -- added check for $ENV{TEMP}
    This fixes issue seen under Cygwin

1.007 Thu Nov 15 21:30:00 EST 2012

  * At last, the "Voila" release :)
  * Small change to __DIE__ and __WARN__ signal handling for spawn method

1.006 Thu Nov 15 03:30:00 EST 2012

  * Added description section to MCE::Signal's Perl doc
  * Do not set trap on __DIE__ and __WARN__ inside MCE::Signal
  * Localized __DIE__ and __WARN__ handlers inside MCE instead
  * Clarify the use of threads in documentation

1.005 Tue Nov 13 06:11:00 EST 2012

  * Removed underscore from package globals in MCE::Signal
  * Optimized _worker_read_handle method in MCE
  * Updated files under examples/tbray/

1.004 Mon Nov 12 01:50:00 EST 2012

  * Updated examples/mce_usage.readme
  * Updated examples/
  * Added examples/tbray/README
  * Added examples/tbray/
  * Added examples/tbray/
  * Added examples/tbray/
  * Added examples/tbray/
  * Added examples/tbray/ (../ moved here)
  * Added examples/tbray/

1.003 Sat Nov 10 12:55:00 EST 2012

  * Updated README
  * Updated images/06_Shared_Sockets.gif
  * Updated images/10_Scaling_Pings.gif
  * Added   images/11_SNMP_Collection.gif
  * Small update to MCE::Signal

1.002 Thu Nov 08 01:13:10 EST 2012

  * Renamed continue method to next

1.001 Wed Nov 07 23:58:20 EST 2012

  * Added perl-MCE.spec to trunk
  * Added CREDITS
  * Added 3 new methods to continue, last, and exit
  * Both foreach & forchunk now call run(1, {...}) to auto-shutdown workers

1.000 Mon Nov 05 10:00:00 EST 2012

  * First release