NAME
IPC::Shareable - Use shared memory backed variables across processes
SYNOPSIS
use IPC::Shareable qw(:lock);
tie my %hash, 'IPC::Shareable', OPTIONS;
tie my @array, 'IPC::Shareable', OPTIONS;
tie my $scalar, 'IPC::Shareable', OPTIONS;
# Get SYSV shared memory specifications of the system (if available)
my $href = IPC::Shareable::sysv_info();
# Lock, make changes, unlock
tied(VARIABLE)->lock;
# Do something with the variable
tied(VARIABLE)->unlock;
# Non-blocking lock attempt
tied(VARIABLE)->lock(LOCK_SH|LOCK_NB)
or print "Resource unavailable\n";
# Lock with a code reference, which will auto-unlock when the block finishes
tied(VARIABLE->lock(sub { print "hello!\n"; });
# Get the shared memory segment and semaphore objects directly
my $segment = tied(VARIABLE)->seg;
my $semaphore = tied(VARIABLE)->sem;
# Remove the shared memory segment and semaphore directly
tied(VARIABLE)->remove;
# Manual cleanup procedures
IPC::Shareable::clean_up;
IPC::Shareable::clean_up_all;
IPC::Shareable::clean_up_protected;
# Ensure only one instance of a script can be run at any time
IPC::Shareable->singleton('UNIQUE SCRIPT LOCK STRING');
# Get the actual IPC::Shareable tied object you can make method calls on
# instead of using the tied object like the examples above
my $knot = tied(VARIABLE); # Dereference first if using a tied reference
# ...or get the knot at inception
my $knot = tie my VARIABLE, 'IPC::Shareable', OPTIONS;
my $sysv_info_href = $knot->sysv_info;
DESCRIPTION
IPC::Shareable allows you to tie a variable to shared memory making it easy to share the contents of that variable with other Perl processes and scripts.
Scalars, arrays, hashes and even objects can be tied. The variable being tied may contain arbitrarily complex data structures - including references to arrays, hashes of hashes, etc.
Note: When using nested data structures, each nested structure utilizes an additional shared memory segment. The entire structure is not squashed into a single segment. See "DATA AND SEGMENT MAPPING" for details.
The association between variables in distinct processes is provided by GLUE (aka "key"). This is any arbitrary string or integer that serves as a common identifier for data across process space. Hence the statement:
tie my %hash, 'IPC::Shareable', { key => 'GLUE STRING', create => 1 };
...in program one and the statement
tie my %thing, 'IPC::Shareable', { key => 'GLUE STRING' };
...in program two will create and bind %hash the shared memory in program one and bind it to %thing in program two.
There is no pre-set limit to the number of processes that can bind to data; nor is there a pre-set limit to the complexity of the underlying data of the tied variables. The amount of data that can be shared within a single bound variable is limited by the system's maximum size for a shared memory segment (the exact value is system-dependent).
The bound data structures are all linearized (using Raphael Manfredi's Storable module or optionally JSON) before being slurped into shared memory. Upon retrieval, the original format of the data structure is recovered. Semaphore flags can be used for locking data between competing processes.
OPTIONS
a.k.a "attributes"
Options are specified by passing a reference to a hash as the third argument to the tie() function that enchants a variable. We also call these "Attributes"
The following fields are recognized in the options hash:
key
key is the GLUE that is a direct reference to the shared memory segment that's to be tied to the variable.
If this option is missing, we'll default to using IPC_PRIVATE. This default key will not allow sharing of the variable between processes.
The key can be specified as:
A text string (internally, a 32-bit CRC of the string is used as the key)
A hex string (e.g.
'0xDEADBEEF'), used as-is as the integer keyAn integer (e.g.
1234), used as-is as the integer key
Default: IPC_PRIVATE
create
create is used to control whether the process creates a new shared memory segment or not. If create is set to a true value, IPC::Shareable will create a new binding associated with GLUE as needed. If create is false, IPC::Shareable will not attempt to create a new shared memory segment associated with GLUE. In this case, a shared memory segment associated with GLUE must already exist or we'll croak().
Defult: false
exclusive
If exclusive field is set to a true value, we will croak() if the data binding associated with GLUE already exists. If set to a false value, calls to tie() will succeed even if a shared memory segment associated with GLUE already exists.
See "graceful" for a silent, non-exception exit if a second process attempts to obtain an in-use exclusive segment.
Default: false
graceful
If exclusive is set to a true value, we normally croak() if a second process attempts to obtain the same shared memory segment. Set graceful to true and we'll exit silently and gracefully. This option does nothing if exclusive isn't set.
Useful for ensuring only a single process is running at a time.
Default: false
warn
When set to a true value, graceful will output a warning if there are process collisions.
Default: false
mode
The mode argument is an octal number specifying the access permissions when a new data binding is being created. These access permission are the same as file access permissions in that 0666 is world readable and writable, 0600 is readable only by the effective UID of the process creating the shared variable, etc.
Default: 0666 (world readable and writeable)
size
This field may be used to specify the size of the shared memory segment allocated.
Note: Each nested data structure requires a new shared memory segment. The size attribute is applied to the first, and all subsequent segments created, and does not reflect the overall size of memory to be used.
The maximum size we allow by default is ~1GB. See the "limit" option to override this default.
Default: IPC::Shareable::SHM_BUFSIZ() (ie. 65536)
protected
If set, the clean_up() and clean_up_all() routines will not remove the segments or semaphores related to the tied object.
Set this to a specific integer so we can pass the value to any child objects created under the main one.
To clean up protected objects, call (tied %object)->clean_up_protected(integer), where 'integer' is the value you set the protected option to. You can call this cleanup routine in the script you created the segment, or anywhere else, at any time.
Default: 0
limit
This field will allow you to set a segment size larger than the default maximum which is 1,073,741,824 bytes (approximately 1 GB). If set, we will croak() if a size specified is larger than the maximum. If it's set to a false value, we'll croak() if you send in a size larger than the total system RAM.
Default: true
enforced_locking
This attribute will allow you to enforce locks that you set, instead of them being simply advisory.
Use with violated_lock_warn to emit a warning when a lock collision has occurred.
Default: true
violated_lock_warn
When enforced_locking is enabled, and this attribute is set to true, we will emit a warning when an exclusive lock collision has occurred. The warning will include the UUID of the object that caused the violation, and the segment ID that the violation occurred against.
Default: false
destroy
If set to a true value, the shared memory segment underlying the data binding will be removed when the process that initialized the shared memory segment exits (gracefully)[1].
Only those memory segments that were created by the current process will be removed.
Use this option with care. In particular you should not use this option in a program that will fork after binding the data. On the other hand, shared memory is a finite resource and should be released if it is not needed.
NOTE: If the segment was created with its "protected" attribute set, it will not be removed upon program completion, even if destroy is set.
Default: false
tidy
For long running processes, set this to a true value to clean up unneeded segments from nested data structures. Comes with a slight 2% performance hit.
Default: true
serializer
By default, we use Storable as the data serializer when writing to or reading from the shared memory segments we create. For cross-platform and cross-language purposes, you can optionally use JSON for this task.
Send in either json or storable as the value to use the respective serializer.
Default: storable
Default Option Values
Default values for options are:
key => IPC_PRIVATE, # 0
create => 0,
exclusive => 0,
mode => 0666,
size => IPC::Shareable::SHM_BUFSIZ(), # 65536
protected => 0,
limit => 1,
destroy => 0,
graceful => 0,
warn => 0,
tidy => 1,
serializer => 'storable',
enforced_locking => 1,
violated_lock_warn => 0,
METHODS
new
This new() call is not necessary and is a simple wrapper around tie(). It is capable only of returning a tied hash reference object.
Instantiates and returns a reference to a hash backed by shared memory.
my $href = IPC::Shareable->new(key => "testing", create => 1);
$href=>{a} = 1;
# Call tied() on the dereferenced variable to access object methods
# and information
tied(%$href)->shm_count;
Parameters:
Hash, Optional: See the "OPTIONS" section for a list of all available options. Most often, you'll want to send in the key and create options.
It is possible to get a reference to an array or scalar as well. Simply send in either var = > 'ARRAY' or var => 'SCALAR' to do so.
Return: A reference to a hash (or array or scalar) which is backed by shared memory.
uuid
Returns the UUID of the object.
singleton($glue, $warn)
Class method that ensures that only a single instance of a script can be run at any given time.
Parameters:
$glue
Mandatory, String: The key/glue that identifies the shared memory segment.
$warn
Optional, Bool: Send in a true value to have subsequent processes throw a warning that there's been a shared memory violation and that it will exit.
Default: false
Note: See Script::Singleton. That library implements singleton for a script with a simple use line.
shm_count
Returns the number of instantiated shared memory segments that currently exist on the system. This isn't precise; it simply does a wc -l line count on your system's ipcs -m call. It is guaranteed though to produce reliable results.
Return: Integer
shm_segments
my $ipc_shareable_segments = IPC::Shareable->shm_segments;
Class/object method. Scans all existing shared memory segments on the system and returns a hash reference mapping the hex key string (e.g. '0xdeadbeef') to the raw literal contents of that segment. Only loads segments that were created by IPC::Shareable.
Segments created with IPC_PRIVATE (key 0x00000000) are skipped because they cannot be looked up by key.
Return: Hash reference where each key is the SHM key in hex format.
Example return:
orphaned: Was created by IPC::Shareable, but is no longer associated with any process. Should be cleaned up.
local_process: 1 if created by the same process this method is being run, and 0 if not.
content: The actual raw content of the shared memory segment.
child_keys: Nested data structures each require their own segment. Keys within this array reference map to child segments.
{
'0x2abc0001' => {
orphaned => 0,
local_process => 1
content => 'IPC::Shareable{"b":"hello","c":{"__ics__":{"child_key_hex":"0x000e1b1d","child_key":"924445","type":"HASH"}},"a":1,"d":{"__ics__":{"child_key_hex":"0x000097af","child_key":"38831","type":"HASH"}}}',
child_keys => [
'0x000e1b1d',
'0x000097af'
],
},
'0x000e1b1d' => {
orphaned => 0,
local_process => 1
content => 'IPC::Shareable{"y":20,"x":10}',
child_keys => [],
},
'0x000097af' => {
orphaned => 0,
local_process => 1
content => 'IPC::Shareable{"p":"foo","q":"bar"}',
child_keys => [],
}
}
orphans
my @orphan_keys = IPC::Shareable->orphans;
# or
my @orphan_keys = $knot->orphans;
Class/object method. Returns a list of hex key strings (e.g. '0xdeadbeef') for all shared memory segments that were created by IPC::Shareable but are no longer associated with any process (i.e. orphaned). Orphaned segments are typically left over from processes that exited without cleaning up (read: crashed).
Return: List of hex key strings.
my @orphans = IPC::Shareable->orphans;
for my $key (@orphans) {
print "Orphaned segment: $key\n";
IPC::Shareable->remove($key);
}
sysv_info
my $sysv_info = IPC::Shareable->sysv_info;
print "Max segment size: $sysv_info->{shmmax}\n";
print "Max segments (system): $sysv_info->{shmmni}\n";
Class method. Returns a hash reference containing the kernel's SysV shared memory configuration parameters for the current platform.
Returns undef if the platform is not supported or no data could be read.
On macOS, reads from sysctl kern.sysv. Example return value:
{
shmmax => 4194304, # Maximum size of a single segment (bytes)
shmmin => 1, # Minimum size of a single segment (bytes)
shmmni => 32, # Maximum number of segments system-wide
shmseg => 8, # Maximum number of segments per process
shmall => 1024, # Maximum total shared memory (pages)
}
On Linux, reads from /proc/sys/kernel/. Example return value:
{
shmmax => 18446744073692774399, # Maximum size of a single segment (bytes)
shmmin => 1, # Minimum size of a single segment (bytes)
shmmni => 4096, # Maximum number of segments system-wide
shmall => 18446744073692774399, # Maximum total shared memory (pages)
}
Note: Linux has no per-process segment limit (shmseg); only the system-wide shmmni applies.
Return: Hash reference, or undef if the platform is not supported or no data could be read.
lock($flags, $code)
Parameters:
$flags
Optional, Integer: See flock() system call for lock flag combinations. If this parameter is emitted, we default to LOCK_EX, an exclusive blocking lock.
$code
Optional, Code reference: If this parameter is sent in, and an exclusive lock is asked for, we will set the lock, execute the subroutine, and then call unlock() on the segment.
Obtains a lock on the shared memory. $flags specifies the type of lock to acquire. If $flags is not specified, an exclusive read/write lock is obtained. Acceptable values for $flags are the same as for the flock() system call.
Returns true on success, and undef on error. For non-blocking calls (see below), the method returns 0 if it would have blocked.
Note: Although the $flags and $code parameters appear positional, you can send in $code without sending in any $flags. When this occurs, $flags will automatically be set to LOCK_EX.
Obtain an exclusive lock like this:
tied(%var)->lock(LOCK_EX); # Same as default
Only one process can hold an exclusive lock on the shared memory at a given time.
Obtain a shared (read) lock:
tied(%var)->lock(LOCK_SH);
Multiple processes can hold a shared (read) lock at a given time. If a process attempts to obtain an exclusive lock while one or more processes hold shared locks, it will be blocked until they have all finished.
Either of the locks may be specified as non-blocking:
tied(%var)->lock( LOCK_EX|LOCK_NB );
tied(%var)->lock( LOCK_SH|LOCK_NB );
A non-blocking lock request will return 0 if it would have had to wait to obtain the lock.
Note that these locks are advisory (just like flock), meaning that all cooperating processes must coordinate their accesses to shared memory using these calls in order for locking to work. See the flock() call for details.
Locks are inherited through forks, which means that two processes actually can possess an exclusive lock at the same time. Don't do that.
The constants LOCK_EX, LOCK_SH, LOCK_NB, and LOCK_UN are available for import using any of the following export tags:
use IPC::Shareable qw(:lock);
use IPC::Shareable qw(:flock);
use IPC::Shareable qw(:all);
Or, just use the flock constants available in the Fcntl module.
See "LOCKING" for further details.
unlock
Removes a lock. Takes no parameters, returns true on success.
This is equivalent of calling shlock(LOCK_UN).
See "LOCKING" for further details.
seg
Called on either the tied variable or the tie object, returns the shared memory segment object currently in use.
See IPC::Shareable::SharedMem documentation for details.
sem
Called on either the tied variable or the tie object, returns the semaphore object related to the memory segment currently in use.
attributes
Retrieves the list of attributes that drive the IPC::Shareable object.
Parameters:
$attribute
Optional, String: The name of the attribute. If sent in, we'll return the value of this specific attribute. Returns undef if the attribute isn't found.
Attributes are the OPTIONS that were used to create the object.
Returns: A hash reference of all attributes if $attributes isn't sent in, the value of the specific attribute if it is.
global_register
Returns a hash reference of hashes of all in-use shared memory segments across all processes. The key is the memory segment ID, and the value is the segment and semaphore objects.
process_register
Returns a hash reference of hashes of all in-use shared memory segments created by the calling process. The key is the memory segment ID, and the value is the segment and semaphore objects.
LOCKING
IPC::Shareable provides methods to implement application-level advisory and enforced locking of the shared data structures. These methods are lock() and unlock(). To use them you must first get the object underlying the tied variable, either by saving the return value of the original call to tie() or by using the built-in tied() function.
Lock and unlock
To lock and subsequently unlock a variable, do this:
my $knot = tie my %hash, 'IPC::Shareable', { %options };
$knot->lock;
$hash{a} = 'foo';
$knot->unlock;
or equivalently, if you've decided to throw away the return of tie():
tie my %hash, 'IPC::Shareable', { %options };
tied(%hash)->lock;
$hash{a} = 'foo';
tied(%hash)->unlock;
This will place an exclusive lock on the data of %hash. You can also get shared locks or attempt to get a lock without blocking.
IPC::Shareable makes the constants LOCK_EX, LOCK_SH, LOCK_UN, and LOCK_NB exportable to your address space with the export tags :lock, :flock, or :all. The values should be the same as the standard flock option arguments.
if (tied(%hash)->lock(LOCK_SH|LOCK_NB)){
print "The value is $hash{a}\n";
tied(%hash)->unlock;
} else {
print "Another process has an exclusive lock.\n";
}
If no argument is provided to lock, it defaults to LOCK_EX.
Enforced write locking
By default, the enforced_locking is set to true, which means that if a tied variable sets a LOCK_EX, all writes from all other processes will fail, regardless of whether the other processes opted in to check locking.
violated_lock_warn Option
Disabled by default, this is an attribute you set on object instantiation. If set to true and another object has a LOCK_EX in place during a write operation, a warning will be emitted by any process that attempts to write to a locked segment.
Important notes
Note that in the background, we perform lock optimization when reading and writing to the shared storage even if the advisory locks aren't being used.
Using the advisory locks can speed up processes that are doing several writes/ reads at the same time.
When using lock() to lock a variable, be careful to guard against signals. Under normal circumstances, IPC::Shareable's END method unlocks any locked variables when the process exits. However, if an untrapped signal is received while a process holds a lock, END will not be called.
This is not a deadlock risk: all semaphore lock operations in IPC::Shareable use the SEM_UNDO flag, which causes the kernel to automatically reverse any semaphore operations when the process exits, regardless of the cause of death (including SIGKILL and hardware faults). Other processes waiting for the lock will be unblocked.
DESTRUCTION
perl(1) will destroy the object underlying a tied variable when then tied variable goes out of scope. Unfortunately for IPC::Shareable, this may not be desirable: other processes may still need a handle on the relevant shared memory segment.
IPC::Shareable therefore provides several options to control the timing of removal of shared memory segments.
destroy Option
As described in "OPTIONS", specifying the destroy option when tie()ing a variable coerces IPC::Shareable to remove the underlying shared memory segment when the process calling tie() exits gracefully.
NOTE: The destruction is handled in an END block. Only those memory segments that are tied to the current process will be removed.
NOTE: If the segment was created with its "protected" attribute set, it will not be removed in the END block, even if destroy is set.
NOTE: The END block only runs on a clean exit (normal program end, die, or exit). It does not run for untrapped signals (SIGTERM, SIGINT, etc.) or for SIGKILL. If your process may be terminated by a signal and you want destroy cleanup to run, install signal handlers that call exit:
$SIG{INT} = $SIG{TERM} = $SIG{HUP} = sub { exit };
This causes the END block to fire on those signals. SIGKILL cannot be caught; any segments left behind by it can be recovered with IPC::Shareable->clean_up_all.
NOTE: Advisory locks (lock()/unlock()) are always released automatically when a process dies, even on SIGKILL, because the underlying semaphore operations use SEM_UNDO. Lock release is therefore not a concern; only shared memory segment data requires the signal handler precaution above.
remove($key)
Parameters:
$key
Optional, see "key" for valid values. Preferably, an integer or a hex string prefixed with 0x.
Note: If the $key parameter is sent in, we will delete that segment only and return immediately thereafter.
tied($var)->remove;
# or
$knot->remove;
# Remove a specific segment by key (can remove non C<IPC::Shareable>
segments). If key is sent in, the caller can be the module or the object.
IPC::Shareable->remove('0xdeadbeef'); # hex string
IPC::Shareable->remove(0xdeadbeef); # hex integer
IPC::Shareable->remove(1234); # integer
tied($var)->remove('Test'); # string
Calling remove() on the object underlying a tie()d variable removes the associated shared memory segments. The segment is removed irrespective of whether it has the destroy option set or not and irrespective of whether the calling process created the segment.
clean_up
IPC::Shareable->clean_up;
# or
tied($var)->clean_up;
# or
$knot->clean_up;
This is a class method that provokes IPC::Shareable to remove all shared memory segments created by the process. Segments not created by the calling process are not removed.
This method will not clean up segments created with the protected option.
clean_up_all
IPC::Shareable->clean_up_all;
# or
tied($var)->clean_up_all;
# or
$knot->clean_up_all
This is a class method that provokes IPC::Shareable to remove all shared memory segments encountered by the process. Segments are removed even if they were not created by the calling process.
This method will not clean up segments created with the protected option.
clean_up_protected($protect_key)
If a segment is created with the protected option, it, nor its children will be removed during calls of clean_up() or clean_up_all().
When setting "protected", you specified a lock key integer. When calling this method, you must send that integer in as a parameter so we know which segments to clean up.
my $protect_key = 93432;
IPC::Shareable->clean_up_protected($protect_key);
# or
tied($var)->clean_up_protected($protect_key;
# or
$knot->clean_up_protected($protect_key)
Parameters:
$protect_key
Mandatory, Integer: The integer protect key you assigned wit the protected option
RETURN VALUES
Calls to tie() that try to implement IPC::Shareable will return an instance of IPC::Shareable on success, and undef otherwise.
DATA AND SEGMENT MAPPING
For simple data (none of the values are references), a single segment is used throughout. However, with nested data, each value that is a reference is stored in its own, separate shared memory segment (the key is auto-generated).
Consider a three-level hash:
$h{a}{b}{c} = 1;
This creates three segments:
Root segment (SysV key 0xABCD)
stored data: { a => <pointer to child key=11111> }
|
v
Child segment (SysV key 11111)
stored data: { b => <pointer to grandchild key=22222> }
|
v
Grandchild segment (SysV key 22222)
stored data: { c => 1 }
Each segment only knows about its direct children. The chain is followed lazily, one level at a time, as you FETCH down into the structure.
When you replace a child with a new reference where the previous value was also a reference, a new segment is created and the new data is stored there. If tidy is enabled (default), the old segment is automatically removed.
When a value that is a reference is deleted from the data, the memory segment that held that data is automatically cleaned up and freed.
Storable
The child knot object (which holds _key, _type, etc.) is frozen in-place inside the parent's serialized byte blob. On thaw, the child knot is reconstructed from those bytes and re-attached to the existing child segment.
JSON
JSON can't serialize blessed objects, so each child pointer is written as an explicit marker:
{ "__ics__" => { type => "HASH", child_key => 11111, child_key_hex => "0x00002b67" } }
The raw JSON in the root segment looks like:
{"a":{"__ics__":{"type":"HASH","child_key":11111,"child_key_hex":"0x00002b67"}}}
The raw JSON in the child segment (key 11111) looks like:
{"b":{"__ics__":{"type":"HASH","child_key":22222,"child_key_hex":"0x000056ce"}}}
Finally, the value in the child is not a reference, so it's stored as literal data:
{"c": 1}
On decode, any __ics__ marker is spotted and a tie with create => 0 is used to re-attach to the existing child segment by that key -- no new segment is created, it simply reconnects.
AUTHOR
Benjamin Sugars <bsugars@canoe.ca>
MAINTAINED BY
Steve Bertrand <steveb@cpan.org>
NOTES
General Notes
- o
-
There is a program called
ipcs(1/8) (andipcrm(1/8)) that is available on at least Solaris and Linux that might be useful for cleaning moribund shared memory segments or semaphore sets produced by bugs in either IPC::Shareable or applications using it.Examples:
# List all semaphores and memory segments in use on the system ipcs -a # List all memory segments and semaphores along with each one's associated process ID ipcs -ap # List just the shared memory segments ipcs -m # List the details of an individual memory segment ipcs -i 12345678 # Remove *all* semaphores and memory segments ipcrm -a - o
-
This version of IPC::Shareable does not understand the format of shared memory segments created by versions prior to
0.60. If you try to tie to such segments, you will get an error. The only work around is to clear the shared memory segments and start with a fresh set. - o
-
Iterating over a hash causes a special optimization if you have not obtained a lock (it is better to obtain a read (or write) lock before iterating over a hash tied to IPC::Shareable, but we attempt this optimization if you do not).
For tied hashes, the
fetch/thawoperation is performed when the first key is accessed. Subsequent key and and value accesses are done without accessing shared memory. Doing an assignment to the hash or fetching another value between key accesses causes the hash to be replaced from shared memory. The state of the iterator in this case is not defined by the Perl documentation. Caveat Emptor.
CREDITS
Thanks to all those with comments or bug fixes, especially
Maurice Aubrey <maurice@hevanet.com>
Stephane Bortzmeyer <bortzmeyer@pasteur.fr>
Doug MacEachern <dougm@telebusiness.co.nz>
Robert Emmery <roberte@netscape.com>
Mohammed J. Kabir <kabir@intevo.com>
Terry Ewing <terry@intevo.com>
Tim Fries <timf@dicecorp.com>
Joe Thomas <jthomas@women.com>
Paul Makepeace <Paul.Makepeace@realprogrammers.com>
Raphael Manfredi <Raphael_Manfredi@pobox.com>
Lee Lindley <Lee.Lindley@bigfoot.com>
Dave Rolsky <autarch@urth.org>
Steve Bertrand <steveb@cpan.org>
SEE ALSO
perltie, Storable, shmget, ipcs, ipcrm and other SysV IPC manual pages.