The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.

Revision history for MCE

1.504 Tue Oct 29 16:00:00 EST 2013

  [BUG FIXES]

  * MCE::Grep, MCE::Map, and MCE::Stream were failing for mce_grep_s,
    mce_map_s, and mce_stream_s when specifying chunk_size => 1.

  [ENHANCEMENTS]

  * Calibrated 'auto' slightly in MCE::Util::_parse_chunk_size.

1.503 Mon Oct 28 17:00:00 EST 2013

  [BUG FIXES]

  * The previous release introduced a bug by removing the line closing
    STDERR and STDOUT prior to workers exiting. The proper fix intended
    is to flush (not close) the handles. Closing them was not the thing
    to do in the first place due to possibly needed by the END block or
    the WARN/DIE handlers initiated from inside the END block.

  [ENHANCEMENTS]

  * Updated README under examples/tbray and examples/matmult.
  * Changed 0.499 to 0.5 inside the yield method.
  * Small cosmetic changes otherwise.

1.502 Mon Oct 21 16:00:00 EST 2013

  [BUG FIXES]

  * The END block for all models will return immediately when called by a
    worker thread or process. Removed the line closing STDERR and STDOUT
    prior to workers exiting. It turns out that workers were calling
    shutdown which is not allowed. This impacted MCE::Flow, MCE::Grep,
    MCE::Loop, MCE::Map, and MCE::Stream.

  * An update was applied to DESTROY in MCE::Queue to address an edge
    case during additional testing across several environments.

  * Added documentation describing the core methods in MCE. I had moved the
    missing sections to another file some time back and totally forgotten
    about them when finalizing on MCE::Core.pod for the 1.5 release.

  [ENHANCEMENTS]

  * All models will croak when specifying an invalid MCE option.

  [NEW FEATURES]

  * One can specify a hash reference for the gather option. Updated
    documentation describing the gather option and the gather method.

1.501 Wed Oct 16 01:20:00 EST 2013

  [BUG FIXES]

  * Addressed an issue with the worker signal "die" handler (#89538). Eval'd
    code should not raise an exception causing the app to die. This is working
    as expected for workers spawned as children.

    For threads, a die called explicitly inside an eval block will cause the
    worker to exit (same as previously). However, an eval 'use MissingModule'
    will no longer cause the thread to die.

  [ENHANCEMENTS]

  * Changelog section uses a new format beginning with this release.

  [NEW FEATURES]

  * Added examples/flow_model.pl for demonstrating MCE::Flow, MCE::Queue, and
    MCE->gather.

1.500 Thu Oct 10 01:00:00 EST 2013

  * MCE 1.5 is backwards compatible with 1.4 and below.

  * The documentation, previously MCE.pod, moved to MCE::Core.pod with the
    examples section placed in MCE::Examples.pod. The MCE.pod file serves
    as an index page for the various documentation.

  * IPC has been enhanced with 8 data channels. Many operations run 3x when
    compared with MCE 1.4.

  * Five models: MCE::Flow, MCE::Grep, MCE::Loop, MCE::Map, and MCE::Stream.

  * Hybrid queues via MCE::Queue allowing for normal and priority queues.

  * MCE::Subs for exporting functions prefixed with mce_; e.g. mce_wid.

  * All public methods can be called directly using the package name
    and method e.g. MCE->wid, MCE->run.

       MCE->new(
          max_workers => 'auto',
          user_func => sub {
             my $wid = MCE->wid;
             MCE->sendto("STDOUT", "Hello from $wid\n");
          }
       );

       MCE->run;

  * Localize the input scalar $_ prior to calling user_func. Folks can
    use $_ for input_data and sequence of numbers. Added section to docs
    explaining DEFAULT INPUT SCALAR.

  * New options (bounds_only, gather, interval, task_name).

    The task_end option can now be specified at the top level.
    See docs for use case.

    Input_data can be specified inside the first task instead of having
    to specify this at the top level. Input_data is ignored when specified
    for tasks other than the first task.

       user_tasks => [{
          input_data => \@list,
          ...
       },{
          ...
       }

  * New public methods
    chunk_id, gather, freeze, thaw, yield, task_name, print, printf, say

  * New example (interval.pl).

  * Optimized the egrep.pl and wc.pl examples. These run much faster.
    Try these out against large log files. Both examples fly.

  * Barrier synchronization update. Two sockets are utilized instead of
    2 lock files. This, now works wonderfully under the Cygwin environment.
    For threading, the removal of 2 lock files increases the number of
    threads allowed from about 1/3rd previously to under 1/2 of ulimit,
    e.g. int(ulimit -n / 2 - 20).

  * Removed the logic for determining MAX_OPEN_FILES and MAX_USER_PROCS.
    MCE no longer has a constraint on max_workers allowed.

  * Code re-factor work. Added a private method _validate_runstate called
    by various methods. Organized the code slightly such as placement of
    methods.

1.415 Mon Jun 17 15:00:00 EST 2013

  * Code-refactor in preparation for the upcoming 1.5 release.
    This completes the 1.4x branch for a very solid and stable release.

  * IPC optimization. Localize $\ and $/ only as needed. Small tweak to
    logic when specifying RS (record separator) for input data.

    Foreach is very communication intensive. Forseq is less so. Chunking
    was made faster by increasing chunk size from 500 to 2500. I've reached
    my goal for forseq by reaching 60,000+. A while back, foreach could
    barely reach 18,000.

       Before and after results with IPC optimization.

                             (Before)      (After)
       Parallel::Loops          600          600
       $mce->foreach         20,000       21,500  (+  1,500)
       $mce->forseq          55,000       64,000  (+  9,000)
       $mce->forchunk       395,000      450,000  (+ 55,000)

  * Updated the process method. Allows sequence to be specified as an
    option. Also allow the hash to be specified as the 1st or 2nd
    argument.

       $mce->process( \@input_array, { options } );
       $mce->process( { options }, [ 1..1000 ] );
       $mce->process( { sequence => [ 20, 40, 2 ] } );

  * The forseq method will now honor chunk_size greater than 1. Updated
    docs with use case.

  * Added CONST short for EXPORT_CONST, e.g. use MCE CONST => 1;

1.414 Sun Jun 16 13:00:00 EST 2013

  * Upstream bug fixes. Mainly chunk_size was not honored when specifying
    user_tasks and sequence/chunk_size for the first task. Bug is with
    not setting abort_msg correctly.

1.413 Mon Jun 10 02:00:00 EST 2013

  * Fixed typo: Changed local @_ to local $@ inside _parse_max_workers
    before evaling.

1.412 Sun Jun 09 14:00:00 EST 2013

  * Updated the main README file and CHANGES under 1.411 below.

  * Added support for multiplication and division as well as
    case insentivity to the private _parse_max_workers method.

       max_workers => 'auto',      ## = MCE::Util::get_ncpu()
       max_workers => 'Auto-1',    ## = MCE::Util::get_ncpu() - 1
       max_workers => 'AUTO + 3',  ## = MCE::Util::get_ncpu() + 3
       max_workers => 'AUTO * 1.5',
       max_workers => 'auto / 1.333',

1.411 Sun Jun 09 02:00:00 EST 2013

  * Reverted the change made in MCE-1.410. That degraded slurp IO on
    large files. This restores slurp IO performance in MCE.

  * Corrected an if statement towards the end of the shutdown method.
    The delay was occurring always when it was met only if MCE was
    launched from inside a non-main thread.

  * Addressed a race condition with barrier synchronization under the
    Cygwin environment. Was not able to reproduce this with other OS's.
    This takes barrier synchronization work to 100%. Delay statements
    for Cygwin were removed around barrier synchronization.

  * Optimized writes to sockets. Arguments passed to print statements
    are concatenated as one big string. Removed local $/ = $LF when
    reading from the queue socket. It's not required there since using
    read and known size. The foreach.pl example (very communication
    intensive) can now do 20000 in 1 second on my Macbook Pro. Updated
    benchmark results inside foreach.pl, forchunk.pl, and forseq.pl.

  * MAX_WORKERS, CHUNK_SIZE, TMP_DIR, FREEZE and THAW can be specified
    when loading the module. FREEZE and THAW allows one to choose an
    alternative serialization module if preferred for your project.

       use Sereal qw(encode_sereal decode_sereal);
       use MCE FREEZE => \&encode_sereal, THAW => \&decode_sereal;

  * Passing EXPORT_CONST => 1 will export 3 constants: SELF, CHUNK, CID
    The "my ($self, $chunk_ref, $chunk_id) = @_" line is not necessary.
    $_[SELF], $_[CHUNK], $_[CID] refers to $_[0], $_[1], $_[2].

  * The MCE::Util module was created. It contains the get_ncpu function.
    This is largely borrowed from Test::Smoke::Util.pm. MCE supports
    'auto' when specifying max_workers. Read doc for other use case.

       max_workers => 'auto';  ## Same as MCE::Util::get_ncpu()

  * Re-factored code in preparation for the upcoming 1.5 release. The
    1.5 release will use just 3 socket pairs versus 4. I did not want
    to include that change in this release. Therefore, I decided to
    take the upstream bug fixes and apply them to the 1.4 base.

    Re-factored documentation by making use of the full 78 character
    width per line. In addition tidbits on the new features mentioned
    above.

  * Updated the main README file under the top level dir.

1.410 Tue May 28 23:30:00 EST 2013

  * Use threads under MSWin32 for 02* and 03* test scripts.
  * Removed sysopen/sysseek/sysread to simplify logic due to negligible
    performance gains over open/seek/read.
  * Minor updates to documentation.

1.409 Sun May 12 22:45:00 EST 2013

  * Croak if user_func is not defined and input_data/sequence is specified.
  * Fix barrier synchronization when running multiple tasks via user_tasks.
  * Updated Perl documentation for easier reading through cpan/metapan.
  * Renamed private method _worker_sequence to _worker_sequence_queue.

1.408 Tue Mar 19 22:00:00 EST 2013

  * Minor tweaks here and there to further increase reliability.
  * Updated the barrier synchronization logic to not stall. Updated the
    perl docs on mixing "sync" with "do" or "sendto" methods.
  * Added new "status" method for the manager process.
  * Added new arguments for MCE::Signal: -no_kill9, -no_sigmsg

1.407 Thu Mar 14 21:00:00 EST 2013

  * This marks a tremendous effort in achieving parity across the board
    from Cygwin to Windows and obviously UNIX. MCE now works beautifully
    under the Cygwin environment including Perl for Windows.
  * Up'ed the maximum workers allowed for both Cygwin and Windows to
    56 (forking) and 80 (threading).
  * Barrier synchronization requires an extra semaphore file, therefore
    the maximum workers allowed under the UNIX environment for threading
    decreased. It shouldn't be a problem as there are many threads already.
  * Addressed an issue with barrier synchronization under Cygwin.
  * Addressed an issue with the die handler for the main worker method
    when threading.
  * Thank you for hanging in there. It took quite some time to get there.
    This is the most stable release thus far for MCE across the board.

1.406 Tue Mar 12 19:00:00 EST 2013

  * Added support for barrier synchronization (via new sync method).
    Added barrier_sync.pl example.
  * Addressed rounding errors with the sequence generator.
    The sequence option now follows a bank-teller queuing model when
    generating numbers. This applies to task 0 only. Other tasks can
    still specify sequence where numbers will be distributed equally
    among workers like before.
  * Optimized the _worker_request_chunk private method.
  * A take 2 on the matrix multiplication examples. This is better
    organized with updated README file to include the script running
    time in the results.

1.405 Mon Mar 04 19:00:00 EST 2013

  * Added strassen_pdl_t.pl in the event folks cannot make use of /dev/shm
    used by the strassen_pdl_s.pl example.
  * Optimized the send method -- workers process immediately after receiving
    data. Updated run times in README for the strassen examples.
  * MCE no longer calls setpgrp by default as of MCE 1.405. There is only
    one reason to call setpgrp, but many reasons not to. The sole reason
    was for MCE to run correctly with Daemon::Control. If needed, one can
    pass the option to MCE::Signal qw(-setpgrp).
  * Return void in the shutdown method (previously was returning $self).
  * Tidy code inside sequence generator.

1.404 Sun Feb 24 13:00:00 EST 2013

  * Added sess_dir method
  * Completed work with matmult/* examples
    Added matmult_pdl_q.pl, removed strassen_pdl_h.pl
    Added strassen_pdl_o/p/q/r/s.pl 
    Added benchmark results from a 32-way box at the end of the readme
  * Removed lines setting max limit for files/procs

1.403 Sun Feb 17 15:00:00 EST 2013

  * Wrap sub PDL::CLONE_SKIP into a no warnings 'redefine' block
    MCE now works with PDL::Parallel::threads without any warnings
  * Added missing examples/matmult/matmult_pdl_n.pl to MANIFEST
  * Refactored strassen examples, memory consumption reduced by > than 50%
  * Added matmult_pdl_o.pl -- uses PDL::Parallel::threads to share matrices
  * Added matmult_pdl_p.pl -- matrix b is read from shared memory, not mmap
  * Added strassen_pdl_n.pl -- additional improvements to memory reduction
  * Added strassen_pdl_h.pl -- shown running with 4 workers (half and half)
  * Re-ran matrix multiplication examples and updated results in README file
  * Added -no_setpgrp option to MCE::Signal.pm
    Ctrl-C does not respond when running /usr/bin/time mce_script.pl
  * Added undef $buffer in a couple of places within MCE.pm
  * Added David Mertens and Adam Sjøgren to CREDITS
  * The send method now checks if sending > total workers after spawning
    not before

1.402 Thr Feb 14 07:30:00 EST 2013

  * Updated matrix multiplication examples including README
  * Added examples/matmult/matmult_pdl_n.pl

1.401 Tue Feb 12 19:00:00 EST 2013

  * Added sub PDL::CLONE_SKIP { 1 } to MCE.pm. Running PDL + MCE threads no
    longer crashes during exiting.
  * Updated matrix multiplication examples. All examples now work under the
    windows environment no matter if threading or forking. Unix is stable as
    well if wanting to use PDL + MCE and use_threads => 1 or 0.
  * Added benchmark results for 2048x2048, 4096x4096, and 8192x8192 to the
    README file under examples/matmult/
  * Updated documentation

1.400 Mon Feb 11 07:00:00 EST 2013

  * Slight optimization in the _do_callback method
  * Added 2 new options: user_args and RS (record separator)
  * Added new send method for sending data to workers after spawning and
    prior to running
  * The sequence option can now take an ARRAY reference
  * Updated documentation on new features
  * Added matrix multiplication examples

1.306 Sat Jan 05 18:00:00 EST 2013

  * Added if statement around setpgrp(0,0). That function is not supported
    under Windows.
  * Updated logic for removing any remaining MCE session directories inside
    MCE::Signal.

1.305 Sat Jan 05 16:00:00 EST 2013

  * Added check for $^S to the DIE handler inside the _worker_main method
  * Added setpgrp(0,0) to MCE::Signal's BEGIN block
  * MCE::Signal points to a _mce_sess_dir hash in the event of a signal,
    will remove the sess_dir(s) as well. This is needed when tmp_dir is
    specified during instantiation and pointing to another location than
    MCE::Signal::tmp_dir.

1.304 Wed Jan 02 22:00:00 EST 2013

  * Added Oliver Gorwits to CREDITS for identifying 2 issues
  * Direct die to CORE::die inside handler if executing an eval
  * Undef $mce_spawned_ref if signal was caught (stop_and_exit)
  * Changed INIT to sub import in MCE.pm

1.303 Tue Jan 01 20:00:00 EST 2013

  * Bump version -- MCE.pm VERSION now matches with META.yml
  * Sorted forchunk, foreach, forseq methods inside MCE.pm
  * Modified if statement in run method
  * Task ID is never undef, therefore removed check inside
    restart_worker and worker_do methods
  * Added 2 package variables to MCE::Signal
    $display_die_with_localtime and $display_warn_with_localtime
  * Completed updates to documentation
  * Update to forseq.pl and seq_demo.pl examples

1.302 Tue Jan 01 07:30:00 EST 2013

  * Fixed validation logic for sequence
  * Updated the sequence generator -- now supports chunking
  * Updated seq_demo.pl example to demo user_tasks, sequence, and
    chunk_size applied together
  * Documentation updates

1.301 Mon Dec 31 15:00:00 EST 2012

  * Emphasis on documentation -- better flow plus additional clarity
  * Minor update to sequence option validation
  * Minor update to included examples

1.300 Mon Dec 31 06:00:00 EST 2012

  * New methods...: chunk_size, restart_worker, task_id, task_wid, tmp_dir
  * New options...: on_post_exit, on_post_run, sequence
  * New examples..: forseq.pl, seq_demo.pl
  * Overhaul to exit method
    Workers can exit or die without impacting the manager process
  * Enabled executable bit for test files
  * Removed localtime output in die and warn handlers
  * All 3 delay options are consistent whether or not user_tasks is specified
  * Removed logic around total_ended count -- replaced with new exit logic
  * Code refactoring plus documentation updates
  * Added LICENSE file

1.201 Fri Dec 21 00:00:00 EST 2012

  * Added MCE.pod -- moved documentation from MCE.pm to pod file
  * Added missing use strict/warnings to test scripts
  * Default to 1 for chunk_size and max_workers if not specified
  * Test::More is not a requirement to run MCE, only for building
  * Changed the format for the change log file

1.200 Thu Dec 20 00:00:00 EST 2012

  * Added new user_tasks option
  * Added space between method name and left-paren for header lines in POD
  * Remove not-needed BSD::Resource and forks inside BEGIN/INIT blocks

1.106 Wed Dec 19 05:00:00 EST 2012

  * Added t/pod-coverage.t
  * Big overhaul of the MCE documentation -- all methods are documented
  * Croak if method suited for a MCE worker is called by the main MCE process
  * Croak if method suited for the main MCE process is called by a MCE worker
  * Updated Makefile.PL to declare the minimum Perl version

1.105 Sun Dec 16 23:00:00 EST 2012

  * Completed code re-factoring
  * Added t/pod.t

1.104 Sun Nov 25 17:00:00 EST 2012

  * Added 1 new example to MCE's Perl documentation
  * Use module::method name versus constant symbol when calling _croak
  * Croak if session directory is not writeable inside MCE::spawn
  * Renamed _mce_id to _mce_sid (met to be spawn id actually)
  * Re-calibrated maximum workers allowed

1.103 Fri Nov 23 13:00:00 EST 2012

  * Added writeable check on /dev/shm
  * Croak if tmp dir is not writeable inside MCE::Signal::import

1.102 Thu Nov 22 13:00:00 EST 2012

  * Woohoot !!! MCE now passes with Perl 5.17.x
  * Added Copying file -- same as in Perl

1.101 Wed Nov 21 16:00:00 EST 2012

  * Shifted white space to the left for code blocks inside documentation

1.100 Wed Nov 21 10:00:00 EST 2012

  * Completed optimization and shakeout for MCE's existing API
  * File handles are cached when calling sendto and appending to a file
  * The sendto method now supports multiple arguments -- see perldoc
  * Added new option: flush_file

1.008 Sat Nov 17 23:00:00 EST 2012

  * Update on __DIE__ and __WARN__ handling in MCE. This addresses the 
    unreferenced scalars seen in packaging logs at activestate.com for
    Perl under Windows: http://code.activestate.com/ppm/MCE/
  * Update t/01_load_signal_arg.t -- added check for $ENV{TEMP}
    This fixes issue seen under Cygwin

1.007 Thu Nov 15 21:30:00 EST 2012

  * At last, the "Voila" release :)
  * Small change to __DIE__ and __WARN__ signal handling for spawn method

1.006 Thu Nov 15 03:30:00 EST 2012

  * Added description section to MCE::Signal's Perl doc
  * Do not set trap on __DIE__ and __WARN__ inside MCE::Signal
  * Localized __DIE__ and __WARN__ handlers inside MCE instead
  * Clarify the use of threads in documentation

1.005 Tue Nov 13 06:11:00 EST 2012

  * Removed underscore from package globals in MCE::Signal
  * Optimized _worker_read_handle method in MCE
  * Updated files under examples/tbray/

1.004 Mon Nov 12 01:50:00 EST 2012

  * Updated examples/mce_usage.readme
  * Updated examples/wide_finder.pl
  * Added examples/tbray/README
  * Added examples/tbray/tbray_baseline1.pl
  * Added examples/tbray/tbray_baseline2.pl
  * Added examples/tbray/wf_mce1.pl
  * Added examples/tbray/wf_mce2.pl
  * Added examples/tbray/wf_mce3.pl (../wide_finder.pl moved here)
  * Added examples/tbray/wf_mmap.pl

1.003 Sat Nov 10 12:55:00 EST 2012

  * Updated README
  * Updated images/06_Shared_Sockets.gif
  * Updated images/10_Scaling_Pings.gif
  * Added   images/11_SNMP_Collection.gif
  * Small update to MCE::Signal

1.002 Thu Nov 08 01:13:10 EST 2012

  * Renamed continue method to next

1.001 Wed Nov 07 23:58:20 EST 2012

  * Added perl-MCE.spec to trunk
    http://code.google.com/p/many-core-engine-perl/source/browse/trunk/
  * Added CREDITS
  * Added 3 new methods to MCE.pm: continue, last, and exit
  * Both foreach & forchunk now call run(1, {...}) to auto-shutdown workers

1.000 Mon Nov 05 10:00:00 EST 2012

  * First release