Redis::CappedCollection - Provides fixed size (determined by 'maxmemory' Redis server setting) collections with FIFO data removal.
This documentation refers to Redis::CappedCollection version 1.10
Redis::CappedCollection
use 5.010; use strict; use warnings; #-- Common use Redis::CappedCollection qw( $DEFAULT_SERVER $DEFAULT_PORT ); my $server = $DEFAULT_SERVER.':'.$DEFAULT_PORT; my $coll = Redis::CappedCollection->create( redis => { server => $server } ); # Insert new data into collection my $list_id = $coll->insert( 'Some List_id', 'Some Data_id', 'Some data' ); # Change the element of the list with the ID $list_id $updated = $coll->update( $list_id, $data_id, 'New data' ); # Get data from a list with the ID $list_id @data = $coll->receive( $list_id ); # or to obtain the data ordered from the oldest to the newest while ( my ( $list_id, $data ) = $coll->pop_oldest ) { say "List '$list_id' had '$data'"; }
A brief example of the Redis::CappedCollection usage is provided in "An Example" section.
The data structures used by Redis::CappedCollection on Redis server are explained in "CappedCollection data structure" section.
Redis::CappedCollection module provides fixed sized collections that have a auto-FIFO age-out feature.
The collection consists of multiple lists containing data items ordered by time. Each list must have an unique ID within the collection and each data item has unique ID within its list.
Automatic data removal (when size limit is reached) may remove the oldest item from any list.
Collection size is determined by 'maxmemory' Redis server setting.
Main features of the package are:
Support creation of capped collection, status monitoring, updating the data set, obtaining consistent data from the collection, automatic data removal, error reporting.
Simple API for inserting and retrieving data and for managing collection.
Capped collections are fixed-size collections that have an auto-FIFO age-out feature based on the time of the inserted data. When collection size reaches memory limit, the oldest data elements are removed automatically to provide space for the new elements.
The lists in capped collection store their data items ordered by item time.
To insert a new data item into the capped collection, provide list ID, data ID, data and optional data time (current time is used if not specified). If there is a list with the given ID, the data is inserted into the existing list, otherwise the new list is created automatically.
You may update the existing data in the collection, providing list ID, data ID and optional data time. If no time is specified, the updated data will keep its existing time.
Once the space is fully utilized, newly added data will replace the oldest data in the collection.
Limits are specified when the collection is created. Collection size is determined by 'maxmemory' redis server settings.
The package includes the utilities to dump and restore the collection: dump_collection, restore_collection .
None by default.
Additional constants are available for import, which can be used to define some type of parameters.
These are the defaults:
Default Redis local server: 'localhost'.
'localhost'
Default Redis server port: 6379.
Default socket timeout for connection, number of seconds: 0.1 .
Default socket timeout for read and write operations, number of seconds: 1.
Namespace name used keys on the Redis server: 'C'.
'C'
Minimum and maximum memory reserve limits based on 'maxmemory' configuration of the Redis server.
Not used when 'maxmemory' = 0 (not set in the redis.conf).
'maxmemory'
The following values are used by default:
$MIN_MEMORY_RESERVE = 0.05; # 5% $MAX_MEMORY_RESERVE = 0.5; # 50%
Number of additional elements to delete from collection during cleanup procedure when collection size exceeds 'maxmemory'.
Default 100 elements. 0 means no minimal cleanup required, so memory cleanup will be performed only to free up sufficient amount of memory.
Current data structure version - 3.
More details about error codes are provided in "DIAGNOSTICS" section.
Possible error codes:
$E_NO_ERROR
-1000 - No error
$E_MISMATCH_ARG
-1001 - Invalid argument.
Thrown by methods when there is a missing required argument or argument value is invalid.
$E_DATA_TOO_LARGE
-1002 - Data is too large.
$E_NETWORK
-1003 - Error in connection to Redis server.
$E_MAXMEMORY_LIMIT
-1004 - Command not allowed when used memory > 'maxmemory'.
This means that the command is not allowed when used memory > maxmemory in the redis.conf file.
maxmemory
$E_MAXMEMORY_POLICY
-1005 - Redis server have incompatible maxmemory-policy setting.
maxmemory-policy
Thrown when Redis server have incompatible maxmemory-policy setting in redis.conf.
$E_COLLECTION_DELETED
-1006 - Collection elements was removed prior to use.
This means that the system part of the collection was removed prior to use.
$E_REDIS
-1007 - Redis error message.
This means that other Redis error message detected.
$E_DATA_ID_EXISTS
-1008 - Attempt to add data with an existing ID
This means that you are trying to insert data with an ID that is already in the data list.
$E_OLDER_THAN_ALLOWED
-1009 - Attempt to add outdated data
This means that you are trying to insert the data with the time older than the time of the oldest element currently stored in collection.
$E_NONEXISTENT_DATA_ID
-1010 - Attempt to access the elements missing in the collection.
This means that you are trying to update data which does not exist.
$E_INCOMP_DATA_VERSION
-1011 - Attempt to access the collection with incompatible data structure, created by an older or newer version of this module.
$E_REDIS_DID_NOT_RETURN_DATA
-1012 - The Redis server did not return data.
Check the settings in the file redis.conf.
$E_UNKNOWN_ERROR
-1013 - Unknown error.
Possibly you should modify the constructor parameters for more intense automatic memory release.
create( redis => $server, name => $name, ... )
Create a new collection on the Redis server and return an Redis::CappedCollection object to access it. Must be called as a class method only.
The create creates and returns a Redis::CappedCollection object that is configured to work with the default settings if the corresponding arguments were not given.
create
redis argument can be either an existing object of Redis class (which is then used for all communication with Redis server) or a hash reference used to create a new internal Redis object. See documentation of Redis module for details.
redis
create takes arguments in key-value pairs.
This example illustrates a create() call with all the valid arguments:
create()
my $coll = Redis::CappedCollection->create( redis => { server => "$server:$port" }, # Redis object # or hash reference to parameters to create a new Redis object. name => 'Some name', # Redis::CappedCollection collection name. cleanup_bytes => 50_000, # The minimum size, in bytes, # of the data to be released when performing memory cleanup. cleanup_items => 1_000, # The minimum number of the collection # elements to be realesed when performing memory cleanup. max_list_items => 0, # Maximum list items limit max_datasize => 1_000_000, # Maximum size, in bytes, of the data. # Default 512MB. older_allowed => 0, # Allow adding an element to collection that's older # than the last element removed from collection. # Default 0. check_maxmemory => 1, # Controls if collection should try to find out maximum # available memory from Redis. # In some cases Redis implementation forbids such request, # but setting 'check_maxmemory' to false can be used # as a workaround. memory_reserve => 0.05, # Reserve coefficient of 'maxmemory'. # Not used when 'maxmemory' == 0 (it is not set in the redis.conf). # When you add or modify the data trying to ensure # reserve of free memory for metadata and bookkeeping. reconnect_on_error => 1, # Controls ability to force re-connection with Redis on error. # Boolean argument - default is true and conservative_reconnect is true. connection_timeout => $DEFAULT_CONNECTION_TIMEOUT, # Socket timeout for connection, # number of seconds (can be fractional). # NOTE: Changes external socket configuration. operation_timeout => $DEFAULT_OPERATION_TIMEOUT, # Socket timeout for read and write operations, # number of seconds (can be fractional). # NOTE: Changes external socket configuration. );
The redis and name arguments are required. Do not use the symbol ':' in name.
name
':'
The following examples illustrate other uses of the create method:
my $redis = Redis->new( server => "$server:$port" ); my $coll = Redis::CappedCollection->create( redis => $redis, name => 'Next collection' ); my $next_coll = Redis::CappedCollection->create( redis => $coll, name => 'Some name' );
An error exception is thrown (croak) if an argument is not valid or the collection with same name already exists.
croak
open( redis => $server, name => $name, ... )
Example:
my $redis = Redis->new( server => "$server:$port" ); my $coll = Redis::CappedCollection::open( redis => $redis, name => 'Some name' );
Create a Redis::CappedCollection object to work with an existing collection (created by "create"). It must be called as a class method only.
open takes optional arguments. These arguments are in key-value pairs. Arguments description is the same as for "create" method.
open
The redis and name arguments are mandatory.
The open creates and returns a Redis::CappedCollection object that is configured to work with the default settings if the corresponding arguments are not given.
If redis argument is not a Redis object, a new connection to Redis is established using passed hash reference to create a new Redis object.
An error exception is thrown (croak) if an argument is not valid.
An exception is thrown (croak) if any method argument is not valid or if a required argument is missing.
ATTENTION: In the Redis module the synchronous commands throw an exception on receipt of an error reply, or return a non-error reply directly.
Get collection name attribute (collection ID). The method returns the current value of the attribute. The name attribute value is used in the constructor.
Existing Redis object or a hash reference with parameters to create a new one.
Controls ability to force re-connection with Redis on error.
Controls socket timeout for Redis server connection, number of seconds (can be fractional).
NOTE: Changes external socket configuration.
Controls socket timeout for Redis server read and write operations, number of seconds (can be fractional).
Accessor for cleanup_bytes attribute - The minimum size, in bytes, of the data to be released when performing memory cleanup. Default 0.
cleanup_bytes
The cleanup_bytes attribute is designed to reduce the release of memory operations with frequent data changes.
The cleanup_bytes attribute value can be provided to "create". The method returns and sets the current value of the attribute.
The cleanup_bytes value must be less than or equal to 'maxmemory'. Otherwise an error exception is thrown (croak).
The minimum number of the collection elements to be realesed when performing memory cleanup. Default 100.
The cleanup_items attribute is designed to reduce number of times collection cleanup takes place. Setting value too high may result in unwanted delays during operations with Redis.
cleanup_items
The cleanup_items attribute value can be used in the constructor. The method returns and sets the current value of the attribute.
Maximum list items limit.
Default 0 means that number of list items not limited.
The max_list_items attribute value can be used in the constructor. The method returns and sets the current value of the attribute.
max_list_items
Accessor for the max_datasize attribute.
max_datasize
The method returns the current value of the attribute if called without arguments.
Non-negative integer value can be used to specify a new value to the maximum size of the data introduced into the collection (methods "insert" and "update").
The max_datasize attribute value is used in the constructor and operations data entry on the Redis server.
The constructor uses the smaller of the values of 512MB and 'maxmemory' limit from a redis.conf file.
Accessor for the older_allowed attribute which controls if adding an element that is older than the last element removed from collection is allowed. Default is 0 (not allowed).
older_allowed
0
The method returns the current value of the attribute. The older_allowed attribute value is used in the constructor.
Accessor for the memory_reserve attribute which specifies the amount of additional memory reserved for metadata and bookkeeping. Default 0.05 (5%) of 'maxmemory'. Not used when 'maxmemory' == 0 (it is not set in the redis.conf).
memory_reserve
0.05
Valid values must be between $MIN_MEMORY_RESERVE and $MAX_MEMORY_RESERVE.
$MIN_MEMORY_RESERVE
$MAX_MEMORY_RESERVE
The method returns the current value of the attribute. The memory_reserve attribute value is used in the constructor.
Get code of the last error.
See the list of supported error codes in "DIAGNOSTICS" section.
insert( $list_id, $data_id, $data, $data_time )
$list_id = $coll->insert( 'Some List_id', 'Some Data_id', 'Some data' ); $list_id = $coll->insert( 'Another List_id', 'Data ID', 'More data', Time::HiRes::time() );
Insert data into the capped collection on the Redis server.
Arguments:
$list_id
Mandatory, non-empty string: list ID. Must not contain ':'.
The data will be inserted into the list with given ID, and the list is created automatically if it does not exist yet.
$data_id
Mandatory, non-empty string: data ID, unique within the list identified by $list_id argument.
$data
Data value: a string. Data length should not exceed value of "max_datasize" attribute.
$data_time
Optional data time, a non-negative number. If not specified, the current value returned by time() is used instead. Floating values (such as those returned by Time::HiRes module) are supported to have time granularity of less than 1 second and stored with 4 decimal places.
time()
If collection is set to older_allowed == 1 and $data_time less than time of the last removed element (last_removed_time - see collection_info) then last_removed_time is set to 0. The "older_allowed" attribute value is used in the constructor.
older_allowed == 1
last_removed_time
collection_info
The method returns the ID of the data list to which the data was inserted (value of the $list_id argument).
update( $list_id, $data_id, $data, $new_data_time )
if ( $coll->update( $list_id, $data_id, 'New data' ) ) { say "Data updated successfully"; } else { say "The data is not updated"; }
Updates existing data item.
New data value: a string. Data length should not exceed value of "max_datasize" attribute.
$new_data_time
Optional new data time, a non-negative number. If not specified, the existing data time is preserved.
If the collection is set to older_allowed == 1 and $new_data_time less than time of the last removed element (last_removed_time - see "collection_info") then last_removed_time is set to 0. The "older_allowed" attribute value is used in the constructor.
Method returns true if the data is updated or false if the list with the given ID does not exist or is used an invalid data ID.
Throws an exception on other errors.
upsert( $list_id, $data_id, $data, $data_time )
$list_id = $coll->upsert( 'Some List_id', 'Some Data_id', 'Some data' ); $list_id = $coll->upsert( 'Another List_id', 'Data ID', 'More data', Time::HiRes::time() );
If the list $list_id does not contain data with $data_id, then it behaves like an "insert", otherwise behaves like an "update".
The method returns the ID of the data list to which the data was inserted (value of the $list_id argument) as the "insert" method.
receive( $list_id, $data_id )
my @data = $coll->receive( $list_id ); say "List '$list_id' has '$_'" foreach @data; # or my $list_len = $coll->receive( $list_id ); say "List '$list_id' has '$list_len' item(s)"; # or my $data = $coll->receive( $list_id, $data_id ); say "List '$list_id' has '$data_id'" if defined $data;
If the $data_id argument is not specified or is an empty string:
In a list context, the method returns all the data from the list given by the $list_id identifier.
Method returns an empty list if the list with the given ID does not exist.
In a scalar context, the method returns the length of the data list given by the $list_id identifier.
If the $data_id argument is specified:
The method returns the specified element of the data list. If the data with $data_id ID does not exist, undef is returned.
undef
The method retrieves the oldest data stored in the collection and removes it from the collection.
Returns a list of two elements. The first element contains the identifier of the list from which the data was retrieved. The second element contains the extracted data.
The returned data item is removed from the collection.
Method returns an empty list if the collection does not contain any data.
The following examples illustrate uses of the pop_oldest method:
pop_oldest
while ( my ( $list_id, $data ) = $coll->pop_oldest ) { say "List '$list_id' had '$data'"; }
redis_config_ok( redis => $server )
say 'Redis server config ', $coll->redis_config_ok ? 'OK' : 'NOT OK'; my $redis = Redis->new( server => "$server:$port" ); say 'Redis server config ', Redis::CappedCollection::redis_config_ok( redis => $redis ) ? 'OK' : 'NOT OK' ;
Check whether there is a Redis server config correct, now that the 'maxmemory-policy' setting is 'noeviction'. Returns true if config correct and false otherwise.
It can be called as either the existing Redis::CappedCollection object method or a class function.
If invoked as the object method, redis_config_ok uses the redis attribute from the object as default.
redis_config_ok
If invoked as the class function, redis_config_ok requires mandatory redis argument.
This argument are in key-value pair as described for "create" method.
collection_info( redis => $server, name => $name )
my $info = $coll->collection_info; say 'An existing collection uses ', $info->{cleanup_bytes}, " byte of 'cleanup_bytes', ", $info->{items}, ' items are stored in ', $info->{lists}, ' lists'; # or my $info = Redis::CappedCollection::collection_info( redis => $redis, # or redis => { server => "$server:$port" } name => 'Collection name', );
Get collection information and status. It can be called as either an existing Redis::CappedCollection object method or a class function.
collection_info arguments are in key-value pairs. Arguments description match the arguments description for "create" method:
If invoked as the object method, collection_info, arguments are optional and use corresponding object attributes as defaults.
If called as a class methods, the arguments are mandatory.
Returns a reference to a hash with the following elements:
lists - Number of lists in a collection.
lists
items - Number of data items stored in the collection.
items
oldest_time - Time of the oldest data in the collection. undef if the collection does not contain data.
oldest_time
older_allowed - True if it is allowed to put data in collection that is older than the last element removed from collection.
memory_reserve - Memory reserve coefficient.
cleanup_bytes - The minimum size, in bytes, of the data to be released when performing memory cleanup.
cleanup_items - The minimum number of the collection elements to be realesed when performing memory cleanup.
max_list_items - Maximum list items limit.
data_version - Data structure version.
data_version
last_removed_time - time of the last removed element from collection or 0 if nothing was removed from collection yet.
An error will cause the program to throw an exception (croak) if an argument is not valid or the collection does not exist.
list_info( $list_id )
Get data list information and status.
$list_id must be a non-empty string.
items - Number of data items stored in the data list.
oldest_time - The time of the oldest data in the list. undef if the data list does not exist.
my $oldest_time = $coll->oldest_time;
Get the time of the oldest data in the collection. Returns undef if the collection does not contain data.
An error exception is thrown (croak) if the collection does not exist.
list_exists( $list_id )
say "The collection has '$list_id' list" if $coll->list_exists( 'Some_id' );
Check whether there is a list in the collection with given ID $list_id.
Returns true if the list exists and false otherwise.
collection_exists( redis => $server, name => $name )
say 'The collection ', $coll->name, ' exists' if $coll->collection_exists; my $redis = Redis->new( server => "$server:$port" ); say "The collection 'Some name' exists" if Redis::CappedCollection::collection_exists( redis => $redis, name => 'Some name' );
Check whether there is a collection with given name. Returns true if the collection exists and false otherwise.
If invoked as the object method, collection_exists uses redis and name attributes from the object as defaults.
collection_exists
If invoked as the class function, collection_exists requires mandatory redis and name arguments.
These arguments are in key-value pairs as described for "create" method.
lists( $pattern )
say "The collection has '$_' list" foreach $coll->lists;
Returns an array of list ID of lists stored in a collection. Returns all list IDs matching $pattern if $pattern is not empty. $patten must be a non-empty string.
$pattern
$patten
Supported glob-style patterns:
h?llo matches hello, hallo and hxllo
h?llo
hello
hallo
hxllo
h*llo matches hllo and heeeello
h*llo
hllo
heeeello
h[ae]llo matches hello and hallo, but not hillo
h[ae]llo
hillo
Use '\' to escape special characters if you want to match them verbatim.
'\'
Warning: consider lists as a command that should only be used in production environments with extreme care. Its performance is not optimal for large collections. This command is intended for debugging and special operations. Don't use lists in your regular application code.
In addition, it may cause an exception (croak) if the collection contains a very large number of lists ('Error while reading from Redis server').
'Error while reading from Redis server'
resize( redis => $server, name => $name, ... )
$coll->resize( cleanup_bytes => 100_000 ); my $redis = Redis->new( server => "$server:$port" ); Redis::CappedCollection::resize( redis => $redis, name => 'Some name', older_allowed => 1 );
Use the resize to change the values of the parameters of the collection. It can be called as either the existing Redis::CappedCollection object method or a class function.
resize
If invoked as the object method, resize uses redis and name attributes from the object as defaults. If invoked as the class function, resize requires mandatory redis and name arguments.
It is possible to change the following parameters: older_allowed, cleanup_bytes, cleanup_items, memory_reserve. One or more parameters are required.
Returns the number of completed changes.
An error exception is thrown (croak) if an argument is not valid or the collection does not exist.
drop_collection( redis => $server, name => $name )
$coll->drop_collection; my $redis = Redis->new( server => "$server:$port" ); Redis::CappedCollection::drop_collection( redis => $redis, name => 'Some name' );
Use the drop_collection to remove the entire collection from the redis server, including all its data and metadata.
drop_collection
Before using this method, make sure that the collection is not being used by other customers.
It can be called as either the existing Redis::CappedCollection object method or a class function. If invoked as the class function, drop_collection requires mandatory redis and name arguments. These arguments are in key-value pairs as described for "create" method.
Warning: consider drop_collection as a command that should only be used in production environments with extreme care. Its performance is not optimal for large collections. This command is intended for debugging and special operations. Avoid using drop_collection in your regular application code.
drop_collection mat throw an exception (croak) if the collection contains a very large number of lists ('Error while reading from Redis server').
drop_list( $list_id )
Use the drop_list method to remove the entire specified list. Method removes all the structures on the Redis server associated with the specified list.
drop_list
Method returns true if the list is removed, or false otherwise.
$coll->clear_collection;
Use the clear_collection to remove the entire collection data from the redis server,
clear_collection
Warning: consider clear_collection as a command that should only be used in production environments with extreme care. Its performance is not optimal for large collections. This command is intended for debugging and special operations. Avoid using clear_collection in your regular application code.
clear_collection mat throw an exception (croak) if the collection contains a very large number of lists ('Error while reading from Redis server').
$is_alive = $coll->ping;
This command is used to test if a connection is still alive.
Returns 1 if a connection is still alive or 0 otherwise.
External connections to the server object (eg, C <$redis = Redis->new( ... );>), and the collection object can continue to work after calling ping only if the method returned 1.
If there is no connection to the Redis server (methods return 0), the connection to the server closes. In this case, to continue working with the collection, you must re-create the Redis::CappedCollection object with the "open" method. When using an external connection to the server, to check the connection to the server you can use the $redis->echo( ... ) call. This is useful to avoid closing the connection to the Redis server unintentionally.
$redis->echo( ... )
$coll->quit;
Close the connection with the redis server.
It does not close the connection to the Redis server if it is an external connection provided to collection constructor as existing Redis object. When using an external connection (eg, $redis = Redis->new (...);), to close the connection to the Redis server, call $redis->quit after calling this method.
$redis = Redis->new (...);
$redis->quit
All recognizable errors in Redis::CappedCollection set corresponding value into the "last_errorcode" and throw an exception (croak). Unidentified errors also throw exceptions but "last_errorcode" is not set.
In addition to errors in the Redis module, detected errors are "$E_MISMATCH_ARG", "$E_DATA_TOO_LARGE", "$E_MAXMEMORY_POLICY", "$E_COLLECTION_DELETED", "$E_DATA_ID_EXISTS", "$E_OLDER_THAN_ALLOWED", "$E_NONEXISTENT_DATA_ID", "$E_INCOMP_DATA_VERSION", "$E_REDIS_DID_NOT_RETURN_DATA", "$E_UNKNOWN_ERROR".
The user has the choice:
Use the module methods and independently analyze the situation without the use of "last_errorcode".
Piece of code wrapped in eval {...}; and analyze "last_errorcode" (look at the "An Example" section).
eval {...};
An error exception is thrown with confess if the package variable $DEBUG set to true.
confess
$DEBUG
An example of error handling.
use 5.010; use strict; use warnings; #-- Common --------------------------------------------------------- use Redis::CappedCollection qw( $DEFAULT_SERVER $DEFAULT_PORT $E_NO_ERROR $E_MISMATCH_ARG $E_DATA_TOO_LARGE $E_NETWORK $E_MAXMEMORY_LIMIT $E_MAXMEMORY_POLICY $E_COLLECTION_DELETED $E_REDIS ); # Error handling sub exception { my $coll = shift; my $err = shift; die $err unless $coll; if ( $coll->last_errorcode == $E_NO_ERROR ) { # For example, to ignore return unless $err; } elsif ( $coll->last_errorcode == $E_MISMATCH_ARG ) { # Necessary to correct the code } elsif ( $coll->last_errorcode == $E_DATA_TOO_LARGE ) { # Limit data length } elsif ( $coll->last_errorcode == $E_NETWORK ) { # For example, sleep #sleep 60; # and return code to repeat the operation #return 'to repeat'; } elsif ( $coll->last_errorcode == $E_MAXMEMORY_LIMIT ) { # For example, return code to restart the server #return 'to restart the redis server'; } elsif ( $coll->last_errorcode == $E_MAXMEMORY_POLICY ) { # Correct Redis server 'maxmemory-policy' setting } elsif ( $coll->last_errorcode == $E_COLLECTION_DELETED ) { # For example, return code to ignore #return "to ignore $err"; } elsif ( $coll->last_errorcode == $E_REDIS ) { # Independently analyze the $err } elsif ( $coll->last_errorcode == $E_DATA_ID_EXISTS ) { # For example, return code to reinsert the data #return "to reinsert with new data ID"; } elsif ( $coll->last_errorcode == $E_OLDER_THAN_ALLOWED ) { # Independently analyze the situation } else { # Unknown error code } die $err if $err; } my ( $list_id, $coll, @data ); eval { $coll = Redis::CappedCollection->create( redis => $DEFAULT_SERVER.':'.$DEFAULT_PORT, name => 'Some name', ); }; exception( $coll, $@ ) if $@; say "'", $coll->name, "' collection created."; #-- Producer ------------------------------------------------------- #-- New data eval { $list_id = $coll->insert( 'Some List_id', # list id 123, # data id 'Some data', ); say "Added data in a list with '", $list_id, "' id" ); # Change the "zero" element of the list with the ID $list_id if ( $coll->update( $list_id, 0, 'New data' ) ) { say 'Data updated successfully'; } else { say 'Failed to update element'; } }; exception( $coll, $@ ) if $@; #-- Consumer ------------------------------------------------------- #-- Fetching the data eval { @data = $coll->receive( $list_id ); say "List '$list_id' has '$_'" foreach @data; # or to obtain records in the order they were placed while ( my ( $list_id, $data ) = $coll->pop_oldest ) { say "List '$list_id' had '$data'"; } }; exception( $coll, $@ ) if $@; #-- Utility -------------------------------------------------------- #-- Getting statistics my ( $lists, $items ); eval { my $info = $coll->collection_info; say 'An existing collection uses ', $info->{cleanup_bytes}, " byte of 'cleanup_bytes', ", 'in ', $info->{items}, ' items are placed in ', $info->{lists}, ' lists'; say "The collection has '$list_id' list" if $coll->list_exists( 'Some_id' ); }; exception( $coll, $@ ) if $@; #-- Closes and cleans up ------------------------------------------- eval { $coll->quit; # Before use, make sure that the collection # is not being used by other clients #$coll->drop_collection; }; exception( $coll, $@ ) if $@;
Using currently selected database (default = 0).
CappedCollection package creates the following data structures on Redis:
#-- To store collection status: # HASH Namespace:S:Collection_id # For example: $ redis-cli redis 127.0.0.1:6379> KEYS C:S:* 1) "C:S:Some collection name" # | | | # | +-------+ +------------+ # | | | # Namespace | | # Fixed symbol of a properties hash | # Capped Collection id ... redis 127.0.0.1:6379> HGETALL "C:S:Some collection name" 1) "lists" # hash key 2) "1" # the key value 3) "items" # hash key 4) "1" # the key value 5) "older_allowed" # hash key 6) "0" # the key value 7) "cleanup_bytes" # hash key 8) "0" # the key value 9) "cleanup_items" # hash key 10) "0" # the key value 11) "max_list_items" # hash key 12) "100" # the key value 13) "memory_reserve" # hash key 14) "0.05" # the key value 15) "data_version" # hash key 16) "3" # the key value 17) "last_removed_time" # hash key 18) "0" # the key value ... #-- To store collection queue: # ZSET Namespace:Q:Collection_id # For example: redis 127.0.0.1:6379> KEYS C:Q:* 1) "C:Q:Some collection name" # | | | # | +------+ +-----------+ # | | | # Namespace | | # Fixed symbol of a queue | # Capped Collection id ... redis 127.0.0.1:6379> ZRANGE "C:Q:Some collection name" 0 -1 WITHSCORES 1) "Some list id" ----------+ 2) "1348252575.6651001" | # | | # Score: oldest data_time | # Member: Data List id ... #-- To store CappedCollection data: # HASH Namespace:D:Collection_id:DataList_id # If the amount of data in the list is greater than 1 # ZSET Namespace:T:Collection_id:DataList_id # For example: redis 127.0.0.1:6379> KEYS C:[DT]:* 1) "C:D:Some collection name:Some list id" # If the amount of data in the list is greater than 1 2) "C:T:Some collection name:Some list id" # | | | | # | +-----+ +-------+ + ---------+ # | | | | # Namespace | | | # Fixed symbol of a list of data | | # Capped Collection id | # Data list id ... redis 127.0.0.1:6379> HGETALL "C:D:Some collection name:Some list id" 1) "0" # hash key: Data id 2) "Some stuff" # the key value: Data ... # If the amount of data in the list is greater than 1 redis 127.0.0.1:6379> ZRANGE "C:T:Some collection name:Some list id" 0 -1 WITHSCORES 1) "0" ---------------+ 2) "1348252575.5906" | # | | # Score: data_time | # Member: Data id ...
In order to install and use this package Perl version 5.010 or better is required. Redis::CappedCollection module depends on other packages that are distributed separately from Perl. We recommend the following packages to be installed before installing Redis::CappedCollection :
Const::Fast Digest::SHA1 Mouse Params::Util Redis Try::Tiny
The Redis::CappedCollection module has the following optional dependencies:
Data::UUID JSON::XS Net::EmptyPort Test::Exception Test::NoWarnings Test::RedisServer
If the optional modules are missing, some "prereq" tests are skipped.
The installation of the missing dependencies can either be accomplished through your OS package manager or through CPAN (or downloading the source for all dependencies and compiling them manually).
Redis server version 2.8 or higher is required.
The use of maxmemory-policy all* in the redis.conf file could lead to a serious (and hard to detect) problem as Redis server may delete the collection element. Therefore the Redis::CappedCollection does not work with mode maxmemory-policy all* in the redis.conf.
maxmemory-policy all*
It may not be possible to use this module with the cluster of Redis servers because full name of some Redis keys may not be known at the time of the call the Redis Lua script ('EVAL' or 'EVALSHA' command). So the Redis server may not be able to correctly forward the request to the appropriate node in the cluster.
'EVAL'
'EVALSHA'
We strongly recommend setting maxmemory option in the redis.conf file.
WARN: Not use maxmemory less than for example 70mb (60 connections) for avoid 'used_memory > maxmemory' problem.
Old data with the same time will be forced out in no specific order.
The collection API does not support deleting a single data item.
UTF-8 data should be serialized before passing to Redis::CappedCollection for storing in Redis.
According to Redis documentation:
This module consider that any data sent to the Redis server is a raw octets string, even if it has utf8 flag set. And it doesn't do anything when getting data from the Redis server.
TODO: implement tests for
memory errors (working with internal ROLLBACK commands)
working when maxmemory = 0 (in the redis.conf file)
WARNING: According to initServer() function in redis.c :
initServer()
/* 32 bit instances are limited to 4GB of address space, so if there is * no explicit limit in the user provided configuration we set a limit * at 3 GB using maxmemory with 'noeviction' policy'. This avoids * useless crashes of the Redis instance for out of memory. */
The Redis::CappedCollection module was written, tested, and found working on recent Linux distributions.
There are no known bugs in this package.
Please report problems to the "AUTHOR".
Patches are welcome.
All modules contain detailed information on the interfaces they provide.
The basic operation of the Redis::CappedCollection package module:
Redis::CappedCollection - Object interface to create a collection, addition of data and data manipulation.
Redis::CappedCollection::Util - String manipulation utilities.
Redis - Perl binding for Redis database.
Redis::CappedCollection is hosted on GitHub: https://github.com/TrackingSoft/Redis-CappedCollection
Sergey Gladkov, <sgladkov@trackingsoft.com>
Please use GitHub project link above to report problems or contact authors.
Alexander Solovey
Jeremy Jordan
Vlad Marchenko
Copyright (C) 2012-2016 by TrackingSoft LLC.
This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic at http://dev.perl.org/licenses/artistic.html.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
To install Redis::CappedCollection, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Redis::CappedCollection
CPAN shell
perl -MCPAN -e shell install Redis::CappedCollection
For more information on module installation, please visit the detailed CPAN module installation guide.