The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
Thread-Needs-0.06
River stage zero No dependents
++
/ Thread::Needs

NAME

Thread::Needs - remove unneeded modules from CLONEd memory

SYNOPSIS

    use Thread::Needs;
    use Thread::Needs (Config Thread::Pool);
    no Thread::Needs (Config); # only if previously in -use-

DESCRIPTION

                  *** A note of CAUTION ***

 This module only functions on Perl versions 5.8.0 and later.
 And then only when threads are enabled with -Dusethreads.  It
 is of no use with any version of Perl before 5.8.0 or without
 threads enabled.

                  *************************

In many threaded applications, threads do only very simple things that do not need many (if any) modules. The current threading model however, copies all modules that are available at the moment a thread is started, to the memory of the thread (ensuring an identical environment to the thread from which it was started). Memory that is not being used and which is not being shared between processes. In other words, pretty much wasted memory.

The Thread::Needs module is an experimental module for reducing the memory footprint of threaded Perl applications. It attempts to reduce the amount of memory used by removing all modules, except the ones it is told to keep, from memory in any threads that are started from the thread in which Thread::Needs is invoked.

Please note that this module uses some dirty tricks that may crash your application, specifically with segmentation faults. A segmentation fault is usually an indication that a module got removed when it shouldn't have been. But usually, you will just get an error when a thread starts (or is already running for a while) indicating the absence of a particular module. In that case, you just need to add the name of the module to the list of modules that you need to keep. Beware though that this can be a process that takes a number of iterations, as one module may be using other modules that you are not aware of and which are needed anyway.

Memory savings are greatly dependent on the number and type of modules that have been used when a threads is started. It naturally also depends on the number of threads that are started. Observer memory savings have ranged from 1% (with only a few modules and only 1 thread) upto more than 25% (with 100 threads and some modules), effectively making the difference between having a server go into swap or not on my development machine. Your Mileage May Vary.

CLASS METHODS

There are only 2 class methods that can be called either explicitely or implicitely.

import

 use Thread::Needs qw(Must::Keep::This::Module);

 @notyet = Thread::Needs->import( qw(Must::Keep::This::Module) );

With the "import" class method you can specify additional modules that must not be removed in any threads that are started from the current thread.

The "import" method is called implicitely when parameters are specified with use.

If you call the "import" method explicitely, then the modules that were not already marked to be saved, will be returned. This allows a module to mark modules to be kept, start threads, and then unmark the modules to be kept using the unimport class method.

You should also note that you can call the import() method of a module without having to be sure whether the module is actually loaded. If the "import" method is called without the Thread::Needs module being available, it will execute the UNIVERSAL->import method, effectively turning it into a no-op. So modules can easily call the Thread::Needs->import without having to worry about Thread::Needs being available: if is is available, then you will get the memory savings. If it is not available, then you will not get the memory savings, but it won't break either.

unimport

 no Thread::Needs qw(Must::Not::Keep::This::Module);

 Thread::Needs->unimport( qw(Must::Not::Keep::This::Module) );

With the "unimport" class method you can specify modules that must be removed from the list of modules to be removed. It only makes sense to call with a specific module name of it was previously (implicitely) specified with import.

The "unimport" method is called implicitely when parameters are specified with no.

REQUIRED MODULES

 (none)

EXAMPLES

Some examples of using Thread::Needs.

using Thread::Pool

A simple example when using Thread::Pool:

 use Thread::Pool;
 use Thread::Needs qw(Thread::Pool Thread::Conveyor);

 my $pool = Thread::Pool->new(
  {
   do => sub { warn "Hello $_[0]!\n" },
   workers => 10,
  }
 );

 $pool->job( 'Liz' );
 <>; # Look at "top" when "Hello Liz" is shown
 $pool->job( 'Wendy' );

With the Thread::Needs the memory usage of the above is 7928 KByte. Without it, the memory usage is 9104 KByte. That's over 1 Mbyte of memory saved, about 12%. Well, at least on my (Linux) development machine.

within Thread::Pool

Because the Thread::Pool module internally "knows" it will always need the Thread::Pool and Thread::Conveyor modules, it can tell Thread::Needs itself that the Thread::Pool and Thread::Conveyor modules need to be kept. However, that would mean that any other threads would also keep these modules, which may not be what you want.

The solution is simple: import returns the modules that were not yet marked to be saved, so you can later call the unimport module after the threads have been started.

 my @notyet = Thread::Needs->import( qw(Thread::Pool Thread::Conveyor) );
 $thread = threads->new( \&start_thread );
 Thread::Needs->unimport( @notyet );

Please note that the module itself does not have to do a use Thread::Needs. Because any class inherits from UNIVERSAL.pm, the absence of Thread::Needs will cause the calls to "import" and "unimport" to effectively become no-ops.

CAVEATS

Currently only the namespaces of the modules are zapped. And because the namespaces need to continue to exist because of the random order in which CLONE subroutines are executed (otherwise causing segmentation faults), the namespaces are re-created with just a CLONE stub. Hopefully it will be possible to also have these removed and even other stuff that namespace zapping doesn't remove.

AUTHOR

Elizabeth Mattijsen, <liz@dijkmat.nl>.

Please report bugs to <perlbugs@dijkmat.nl>.

COPYRIGHT

Copyright (c) 2002-2003 Elizabeth Mattijsen <liz@dijkmat.nl>. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

SEE ALSO

threads.