Argon - Simple, fast, and flexible distributed computing
version 0.18
Argon is a distributed processing platform built for Perl. It is designed to offer a simple, flexible, system for quickly building scalable systems with the minimum impact on existing workflows and code structure.
An argon system is controlled by a manager process, whose job it is to schedule tasks among registered workers.
A manager process is started with ar-manager. The manager, workers, and clients must all use the same key (a file containing a key phrase used for encryption).
ar-manager
ar-manager --host localhost --port 8000 --key path/to/secret --verbose 7
Workers are started with ar-worker and must use the same key as the manager.
ar-worker
key
ar-worker --mgr mgrhost:8000 --capacity 8 --key path/to/secret --verbose 7
Connecting to an Argon service is straightforward.
use Argon::Client; use AnyEvent; # Connect my $cv = AnyEvent->condvar; my $ar = Argon::Client->new( host => 'mgrhost', port => 8000, keyfile => 'path/to/key', ready => $cv, retry => 1, ); # Connected! $cv->recv;
A code ref (or any callable reference) may be passed using the ready parameter which will be called once the client is connected. The example uses a condition variable (see "CONDITION VARIABLES" in AnyEvent) to sleep until it is called, making the connection blocking.
ready
Once connected, there are a number of ways to schedule tasks with the manager, the most basic being the "queue" in Argon::Client method.
$client->queue('My::Task::Class', $arg_list, sub { my $reply = shift; my $result = $reply->result; });
There are a couple things to note here. The task class is any class that defines both a new and a run method. The $arg_list will be passed to new. A subroutine is passed which is called when the result is ready. The call to the result method will croak if the task failed.
new
run
$arg_list
result
croak
If retry was set when the client was connected, the task will retry on a logarithmic backoff until the server has the capacity to process the task (see "PREDICTABLE PERFORMANCE DEGREDATION" in Argon).
retry
If the workers were started with the --allow-eval switch, the client may pass code references directly to be evaluated by the workers using the "process" in Argon::Client method.
--allow-eval
local $Argon::ALLOW_EVAL = 1; $client->process(sub { ... }, $arg_list, sub { ... });
One of the key problems with many task queue implementations is the manner in which the system recovers from an interruption in service. In most cases, tasks continue to pile up while the system is unavailable. Once the service is again ready to process tasks, a significant backlog has built up, creating further delay for new tasks added to the queue. This creates a log jam that is often accompanied by incidental service slowdowns that can be difficult to diagnose (for example, overloaded workers clearing out the backlog tie up the database, causing other services to slow down).
Argon prevents these cases by placing the responsibility for the backlog on the client. When the manager determines that the system has reached max capacity, new tasks are rejected until there is room in the queue. From the perspective of the client, there is still a delay in the processing of tasks, but the task queue itself never becomes overloaded and the performance degredation will never overflow onto neighborhing systems as a result.
Another adavantage of having a bounded queue is that clients are aware of the backlog and may report this to callers. System administrators may effectively plan for and respond to increased load by spinning up new servers as needed because they can reliably predict the performance of the system under load given a reliable estimate of the cost imposed by the tasks being performed.
Jeff Ober <sysread@fastmail.fm>
This software is copyright (c) 2017 by Jeff Ober.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.
To install Argon, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Argon
CPAN shell
perl -MCPAN -e shell install Argon
For more information on module installation, please visit the detailed CPAN module installation guide.