Revision history for MCE
1.411 Sun Jun 09 02:00:00 EST 2013
* Reverted the change made in MCE-1.410. Slurp IO is back to MMAP IO
performance on very large files. MCE smokes MMAP during "cold cache"
testing. Otherwise, the same performance when file resides in cache.
* Corrected an if statement towards the end of the shutdown method.
The delay was occurring always when it was met to occur only if
MCE was launched from inside a non-main thread.
* Addressed a race condition with barrier synchronization under the
cygwin environment. Was not able to reproduce this with other OS's.
This finally takes barrier synchronization work to 100%. Delay
statements for cygwin around barrier synchronization were removed.
* Optimization to writes on sockets. The print statement now does
string concatenation versus passing multiple arguments. Also removed
local $/ = $LF when reading from the queue socket. It's not required
there since using read and known size.
./examples/foreach.pl 20000 runs in 1 second on a Macbook Pro
That has been my goal all along and finally reached it. Updated
benchmark results in foreach.pl, forchunk.pl and forseq.pl examples.
* MAX_WORKERS, CHUNK_SIZE, TMP_DIR, FREEZE and THAW can be specified
when loading the module. FREEZE and THAW allows one to choose an
alternative serialization module if preferred for your project.
use Sereal qw(encode_sereal decode_sereal);
use MCE FREEZE => \&encode_sereal, THAW => \&decode_sereal;
* Passing EXPORT_CONST => 1 will export 3 constants: SELF, CHUNK, CID
The "my ($self, $chunk_ref, $chunk_id) = @_" line is not necessary.
This following is possible now to represent $_[0], $_[1], $_[2].
$_[SELF], $_[CHUNK], $_[CID]
* A MCE::Util module was created containing the get_ncpu function.
This is largely borrowed from Test::Smoke::Util.pm. MCE supports
'auto' when specifying max_workers. Read doc for other use case.
max_workers => 'auto'; ## Same as MCE::Util::get_ncpu()
* Re-factored code in preparation for the upcoming 1.5 release. The
1.5 release will use just 3 socket pairs versus 4. I did not want
to include that change to this release. Therefore, I decided to
take the upstream bug fixes and apply them to the 1.4 base.
Re-factored documentation by making use of the full 78 character
width per line. In addition tidbits on the new features mentioned
above.
* Updated the README file.
1.410 Tue May 28 23:30:00 EST 2013
* Use threads under MSWin32 for 02* and 03* test scripts.
* Removed sysopen/sysseek/sysread to simplify logic due to negligible
performance gains over open/seek/read.
* Minor updates to documentation.
1.409 Sun May 12 22:45:00 EST 2013
* Croak if user_func is not defined and input_data/sequence is specified.
* Fix barrier synchronization when running multiple tasks via user_tasks.
* Updated Perl documentation for easier reading through cpan/metapan.
* Renamed private method _worker_sequence to _worker_sequence_queue.
1.408 Tue Mar 19 22:00:00 EST 2013
* Minor tweaks here and there to further increase reliability.
* Updated the barrier synchronization logic to not stall. Updated the
perl docs on mixing "sync" with "do" or "sendto" methods.
* Added new "status" method for the manager process.
* Added new arguments for MCE::Signal: -no_kill9, -no_sigmsg
1.407 Thu Mar 14 21:00:00 EST 2013
* This marks a tremendous effort in achieving parity across the board
from Cygwin to Windows and obviously UNIX. MCE now works beautifully
under the Cygwin environment including Perl for Windows.
* Up'ed the maximum workers allowed for both Cygwin and Windows to
56 (forking) and 80 (threading).
* Barrier synchronization requires an extra semaphore file, therefore
the maximum workers allowed under the UNIX environment for threading
decreased. It shouldn't be a problem as there are many threads already.
* Addressed an issue with barrier synchronization under Cygwin.
* Addressed an issue with the die handler for the main worker method
when threading.
* Thank you for hanging in there. It took quite some time to get there.
This is the most stable release thus far for MCE across the board.
1.406 Tue Mar 12 19:00:00 EST 2013
* Added support for barrier synchronization (via new sync method).
Added barrier_sync.pl example.
* Addressed rounding errors with the sequence generator.
The sequence option now follows a bank-teller queuing model when
generating numbers. This applies to task 0 only. Other tasks can
still specify sequence where numbers will be distributed equally
among workers like before.
* Optimized the _worker_request_chunk private method.
* A take 2 on the matrix multiplication examples. This is better
organized with updated README file to include the script running
time in the results.
1.405 Mon Mar 04 19:00:00 EST 2013
* Added strassen_pdl_t.pl in the event folks cannot make use of /dev/shm
used by the strassen_pdl_s.pl example.
* Optimized the send method -- workers process immediately after receiving
data. Updated run times in README for the strassen examples.
* MCE no longer calls setpgrp by default as of MCE 1.405. There is only
one reason to call setpgrp, but many reasons not to. The sole reason
was for MCE to run correctly with Daemon::Control. If needed, one can
pass the option to MCE::Signal qw(-setpgrp).
* Return void in the shutdown method (previously was returning $self).
* Tidy code inside sequence generator.
1.404 Sun Feb 24 13:00:00 EST 2013
* Added sess_dir method
* Completed work with matmult/* examples
Added matmult_pdl_q.pl, removed strassen_pdl_h.pl
Added strassen_pdl_o/p/q/r/s.pl
Added benchmark results from a 32-way box at the end of the readme
* Removed lines setting max limit for files/procs
1.403 Sun Feb 17 15:00:00 EST 2013
* Wrap sub PDL::CLONE_SKIP into a no warnings 'redefine' block
MCE now works with PDL::Parallel::threads without any warnings
* Added missing examples/matmult/matmult_pdl_n.pl to MANIFEST
* Refactored strassen examples, memory consumption reduced by > than 50%
* Added matmult_pdl_o.pl -- uses PDL::Parallel::threads to share matrices
* Added matmult_pdl_p.pl -- matrix b is read from shared memory, not mmap
* Added strassen_pdl_n.pl -- additional improvements to memory reduction
* Added strassen_pdl_h.pl -- shown running with 4 workers (half and half)
* Re-ran matrix multiplication examples and updated results in README file
* Added -no_setpgrp option to MCE::Signal.pm
Ctrl-C does not respond when running /usr/bin/time mce_script.pl
* Added undef $buffer in a couple of places within MCE.pm
* Added David Mertens and Adam Sjøgren to CREDITS
* The send method now checks if sending > total workers after spawning
not before
1.402 Thr Feb 14 07:30:00 EST 2013
* Updated matrix multiplication examples including README
* Added examples/matmult/matmult_pdl_n.pl
1.401 Tue Feb 12 19:00:00 EST 2013
* Added sub PDL::CLONE_SKIP { 1 } to MCE.pm. Running PDL + MCE threads no
longer crashes during exiting.
* Updated matrix multiplication examples. All examples now work under the
windows environment no matter if threading or forking. Unix is stable as
well if wanting to use PDL + MCE and use_threads => 1 or 0.
* Added benchmark results for 2048x2048, 4096x4096, and 8192x8192 to the
README file under examples/matmult/
* Updated documentation
1.400 Mon Feb 11 07:00:00 EST 2013
* Slight optimization in the _do_callback method
* Added 2 new options: user_args and RS (record separator)
* Added new send method for sending data to workers after spawning and
prior to running
* The sequence option can now take an ARRAY reference
* Updated documentation on new features
* Added matrix multiplication examples
1.306 Sat Jan 05 18:00:00 EST 2013
* Added if statement around setpgrp(0,0). That function is not supported
under Windows.
* Updated logic for removing any remaining MCE session directories inside
MCE::Signal.
1.305 Sat Jan 05 16:00:00 EST 2013
* Added check for $^S to the DIE handler inside the _worker_main method
* Added setpgrp(0,0) to MCE::Signal's BEGIN block
* MCE::Signal points to a _mce_sess_dir hash in the event of a signal,
will remove the sess_dir(s) as well. This is needed when tmp_dir is
specified during instantiation and pointing to another location than
MCE::Signal::tmp_dir.
1.304 Wed Jan 02 22:00:00 EST 2013
* Added Oliver Gorwits to CREDITS for identifying 2 issues
* Direct die to CORE::die inside handler if executing an eval
* Undef $mce_spawned_ref if signal was caught (stop_and_exit)
* Changed INIT to sub import in MCE.pm
1.303 Tue Jan 01 20:00:00 EST 2013
* Bump version -- MCE.pm VERSION now matches with META.yml
* Sorted forchunk, foreach, forseq methods inside MCE.pm
* Modified if statement in run method
* Task ID is never undef, therefore removed check inside
restart_worker and worker_do methods
* Added 2 package variables to MCE::Signal
$display_die_with_localtime and $display_warn_with_localtime
* Completed updates to documentation
* Update to forseq.pl and seq_demo.pl examples
1.302 Tue Jan 01 07:30:00 EST 2013
* Fixed validation logic for sequence
* Updated the sequence generator -- now supports chunking
* Updated seq_demo.pl example to demo user_tasks, sequence, and
chunk_size applied together
* Documentation updates
1.301 Mon Dec 31 15:00:00 EST 2012
* Emphasis on documentation -- better flow plus additional clarity
* Minor update to sequence option validation
* Minor update to included examples
1.300 Mon Dec 31 06:00:00 EST 2012
* New methods...: chunk_size, restart_worker, task_id, task_wid, tmp_dir
* New options...: on_post_exit, on_post_run, sequence
* New examples..: forseq.pl, seq_demo.pl
* Overhaul to exit method
Workers can exit or die without impacting the manager process
* Enabled executable bit for test files
* Removed localtime output in die and warn handlers
* All 3 delay options are consistent whether or not user_tasks is specified
* Removed logic around total_ended count -- replaced with new exit logic
* Code refactoring plus documentation updates
* Added LICENSE file
1.201 Fri Dec 21 00:00:00 EST 2012
* Added MCE.pod -- moved documentation from MCE.pm to pod file
* Added missing use strict/warnings to test scripts
* Default to 1 for chunk_size and max_workers if not specified
* Test::More is not a requirement to run MCE, only for building
* Changed the format for the change log file
1.200 Thu Dec 20 00:00:00 EST 2012
* Added new user_tasks option
* Added space between method name and left-paren for header lines in POD
* Remove not-needed BSD::Resource and forks inside BEGIN/INIT blocks
1.106 Wed Dec 19 05:00:00 EST 2012
* Added t/pod-coverage.t
* Big overhaul of the MCE documentation -- all methods are documented
* Croak if method suited for a MCE worker is called by the main MCE process
* Croak if method suited for the main MCE process is called by a MCE worker
* Updated Makefile.PL to declare the minimum Perl version
1.105 Sun Dec 16 23:00:00 EST 2012
* Completed code re-factoring
* Added t/pod.t
1.104 Sun Nov 25 17:00:00 EST 2012
* Added 1 new example to MCE's Perl documentation
* Use module::method name versus constant symbol when calling _croak
* Croak if session directory is not writeable inside MCE::spawn
* Renamed _mce_id to _mce_sid (met to be spawn id actually)
* Re-calibrated maximum workers allowed
1.103 Fri Nov 23 13:00:00 EST 2012
* Added writeable check on /dev/shm
* Croak if tmp dir is not writeable inside MCE::Signal::import
1.102 Thu Nov 22 13:00:00 EST 2012
* Woohoot !!! MCE now passes with Perl 5.17.x
* Added Copying file -- same as in Perl
1.101 Wed Nov 21 16:00:00 EST 2012
* Shifted white space to the left for code blocks inside documentation
1.100 Wed Nov 21 10:00:00 EST 2012
* Completed optimization and shakeout for MCE's existing API
* File handles are cached when calling sendto and appending to a file
* The sendto method now supports multiple arguments -- see perldoc
* Added new option: flush_file
1.008 Sat Nov 17 23:00:00 EST 2012
* Update on __DIE__ and __WARN__ handling in MCE. This addresses the
unreferenced scalars seen in packaging logs at activestate.com for
Perl under Windows: http://code.activestate.com/ppm/MCE/
* Update t/01_load_signal_arg.t -- added check for $ENV{TEMP}
This fixes issue seen under Cygwin
1.007 Thu Nov 15 21:30:00 EST 2012
* At last, the "Voila" release :)
* Small change to __DIE__ and __WARN__ signal handling for spawn method
1.006 Thu Nov 15 03:30:00 EST 2012
* Added description section to MCE::Signal's Perl doc
* Do not set trap on __DIE__ and __WARN__ inside MCE::Signal
* Localized __DIE__ and __WARN__ handlers inside MCE instead
* Clarify the use of threads in documentation
1.005 Tue Nov 13 06:11:00 EST 2012
* Removed underscore from package globals in MCE::Signal
* Optimized _worker_read_handle method in MCE
* Updated files under examples/tbray/
1.004 Mon Nov 12 01:50:00 EST 2012
* Updated examples/mce_usage.readme
* Updated examples/wide_finder.pl
* Added examples/tbray/README
* Added examples/tbray/tbray_baseline1.pl
* Added examples/tbray/tbray_baseline2.pl
* Added examples/tbray/wf_mce1.pl
* Added examples/tbray/wf_mce2.pl
* Added examples/tbray/wf_mce3.pl (../wide_finder.pl moved here)
* Added examples/tbray/wf_mmap.pl
1.003 Sat Nov 10 12:55:00 EST 2012
* Updated README
* Updated images/06_Shared_Sockets.gif
* Updated images/10_Scaling_Pings.gif
* Added images/11_SNMP_Collection.gif
* Small update to MCE::Signal
1.002 Thu Nov 08 01:13:10 EST 2012
* Renamed continue method to next
1.001 Wed Nov 07 23:58:20 EST 2012
* Added perl-MCE.spec to trunk
http://code.google.com/p/many-core-engine-perl/source/browse/trunk/
* Added CREDITS
* Added 3 new methods to MCE.pm: continue, last, and exit
* Both foreach & forchunk now call run(1, {...}) to auto-shutdown workers
1.000 Mon Nov 05 10:00:00 EST 2012
* First release