Disbatch - a scalable distributed batch processing framework using MongoDB.
version 3.990
"class" defaults to "Disbatch", and the value is then lowercased.
"node" is the hostname.
Anything else is put into $self.
Parameters: type (string, optional)
Returns a Log::Log4perl object.
Parameters: none
Returns a MongoDB::Database object.
Returns a MongoDB::Collection object for collection "nodes".
Returns a MongoDB::Collection object for collection "queues".
Returns a MongoDB::Collection object for collection "tasks".
Loads $self->{config_file} only if $self->{config} is undefined.
$self->{config_file}
$self->{config}
Anything in the config file at startup is static and cannot be changed without restarting disbatchd.
Returns nothing.
Ensures the proper MongoDB indexes are created for tasks, tasks.files, and tasks.chunks collections.
tasks
tasks.files
tasks.chunks
Validates plugins for defined queues.
Clears plugin validation and re-runs validate_plugins().
validate_plugins()
Used by the Disbatch Command Interface to get queue information.
Returns an ARRAY containing HASHes of queue information.
ARRAY
HASH
Throws errors.
Updates the node document with the current timestamp and queues as returned by scheduler_report().
scheduler_report()
Parameters: queue document
Claims a task (sets status to -1 and sets node to hostname) for the given queue.
Returns a task document, or undef if no queued task found.
Parameters: MongoDB::OID object for a task
Sets the task's node to null, status to -2, and update mtime if it has status -1 and this node's hostname.
Returns a task document, or undef if a matching task is not found.
Sets status to -6 for all tasks for this node with status -1 and an mtime of more than 300 seconds ago.
Parameters: queue document, task document
Will fork and exec $self->{config}{task_runner} to start the given task. If the exec fails, it will set threads to 0 for the given queue and call unclaim_task().
$self->{config}{task_runner}
unclaim_task()
Parameters: MongoDB::OID object for a queue or a query operator value or undef, a status or a query operator value or undef, a node or undef.
undef
Counts all tasks for the given $queue_id with given $status and $node.
$queue_id
$status
$node
Used by the below count_* subroutines. If any of the parameters are undef, they will not be added to the query.
count_*
Returns: a non-negative integer, or undef if an error.
Parameters: MongoDB::OID object for a queue or a query operator value or undef
Counts queued (status <= -2), running (status of 0 or -1), running on this node, completed (status >= 1), or all tasks for the given queue (status <= -2).
Parameters: MongoDB::OID object for a queue
Checks config.activequeues if it has entries, and returns 1 if given queue is defined in it or 0 if not. If it does not have entries, checks config.ignorequeues if it has entries, and returns 0 if given queue is defined in it or 1 if not.
config.activequeues
config.ignorequeues
Returns 1 or 0.
Will claim and start as many tasks for each queue as allowed by the current node's maxthreads and each queue's threads.
maxthreads
threads
Parameters: UTF-8 content to store, optional filename to store it as, optional metadata HASH
Stores UTF-8 content in a custom GridFS format that stores data as strings instead of as BinData.
Returns a MongoDB::OID object for the ID inserted in the tasks.files collection.
MongoDB::OID
Parameters: filename or MongoDB::OID object, optional metadata HASH
Gets UTF-8 content from the custom GridFS format. Metadata is only used if given a filename instead of a MongoDB::OID object.
Returns: content string.
Disbatch::Web
Disbatch::Roles
Disbatch::Plugin::Demo
disbatchd
disbatch.pl
task_runner
disbatch-create-users
Ashley Willis <awillis@synacor.com>
Matt Busigin
This software is Copyright (c) 2016 by Ashley Willis.
This is free software, licensed under:
The Apache License, Version 2.0, January 2004
To install Disbatch, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Disbatch
CPAN shell
perl -MCPAN -e shell install Disbatch
For more information on module installation, please visit the detailed CPAN module installation guide.