Helios::Service - base class for services/worker classes in the Helios job processing system
Helios::Service is the base class for all services (aka worker classes) intended to be run by the Helios parallel job processing system. It handles the underlying TheSchwartz job queue system and provides additional methods to handle configuration, job argument parsing, logging, and other functions.
A Helios::Service subclass must implement only one method: the run() method. The run() method will be passed a Helios::Job object representing the job to performed. The run() method should mark the job as completed successfully, failed, or permanently failed (by calling completedJob(), failedJob(), or failedJobPermanent(), respectively) before it ends.
The following 3 methods are used by the underlying TheSchwartz job queuing system to determine what work is to be performed and, a job fails, how it should be retried.
Controls how many times a job will be retried.
Controls how long (in secs) before a failed job will be retried.
These two methods should return the number of times a job can be retried if it fails and the minimum interval between those retries, respectively. If you don't define them in your subclass, they default to zero, and your job(s) will not be retried if they fail.
The work() method is the method called by the underlying TheSchwartz::Worker (which in turn is called by the helios.pl service daemon) to perform the work of a job. Effectively, work() sets up the worker process for the Helios job, and then calls the service subclass's run() method to run it.
The work() method is passed a job object from the underlying TheSchwartz job queue system. The service class is instantiated, and the the job is recast into a Helios::Job object. The service's configuration parameters are read from the system and made available as a hashref via the getConfig() method. The job's arguments are parsed from XML into a Perl hashref, and made available via the job object's getArgs() method. Then the service object's run() method is called, and is passed the Helios::Job object.
Once the run() method has completed the job and returned, work() determines whether the worker process should exit or stay running. If the subclass run() method returns a zero and the worker is OVERDRIVE mode, the worker process will stay running, and work() will be called to setup and run another job. If the run() method returned a nonzero value or the shouldExitOverdrive() returns a true value, the worker process will exit. If OVERDRIVE mode is disabled, the process will also exit.
These accessors will be needed by subclasses of Helios::Service.
get/setConfig() get/setHostname() get/setIniFile() get/setJobType() errstr() debug()
Most of these are handled behind the scenes simply by calling the prep() method.
After calling prep(), calling getConfig() will return a hashref of all the configuration parameters relevant to this service class on this host.
If debug mode is enabled (the HELIOS_DEBUG env var is set to 1), debug() will return a true value, otherwise, it will be false. Some of the Helios::Service methods will honor this value and log extra debugging messages either to the console or the Helios log (helios_log_tb table). You can also use it within your own service classes to enable/disable debugging messages or behaviors.
The new() method doesn't really do much except create an object of the appropriate class. (It can overridden, of course.)
It does set the job type for the object (available via the getJobType() method)
The prep() method is designed to call all the various setup routines needed to get the worker ready to do useful work. It:
Pulls in the contents of the HELIOS_DEBUG and HELIOS_INI env vars, and sets the appropriate instance variables if necessary.
Calls the getConfigFromIni() method to read the appropriate configuration parameters from the INI file.
Calls the getConfigFromDb() method to read the appropriate configuration parameters from the Helios database.
Normally it returns a true value if successful, but if one of the getConfigFrom*() methods throws an exception, that exception will be raised to your calling routine.
The getConfigFromIni() method opens the helios.ini file, grabs global params and config params relevant to the current service class, and returns them in a hash to the calling routine. It also sets the class's internal {config} hashref, so the config parameters are available via the getConfig() method.
Typically job service classes will call this once near the start of processing to pick up any relevant parameters from the helios.ini file. However, calling the prep() method takes care of this for you, and is the preferred method.
The getConfigFromDb() method connects to the Helios database, retrieves config params relevant to the current service class, and returns them in a hash to the calling routine. It also sets the class's internal {config} hasref, so the config parameters are available via the getConfig() method.
Typically service classes will call this once near the start of processing to pick up any relevant parameters from the helios.ini file. However, calling the prep() method takes care of this for you.
There's an important subtle difference between getConfigFromIni() and getConfigFromDb(): getConfigFromIni() erases any previously set parameters from the class's internal {config} hash, while getConfigFromDb() merely updates it. This is due to the way helios.pl uses the methods: the INI file is only read once, while the database is repeatedly checked for configuration updates. For individual service classes, the best thing to do is just call the prep() method; it will take care of things for the most part.
Scans the job queue for jobs that are ready to run. Returns the number of jobs waiting. Only meant for use with the helios.pl service daemon.
Returns a Data::ObjectDriver object for use with the Helios database.
Determine whether or not to exit if OVERDRIVE mode is enabled. The config params will be checked for HOLD, HALT, or OVERDRIVE values. If HALT is defined or HOLD == 1 this method will return a true value, indicating the worker process should exit().
Method to connect to a database. If parameters not specified, uses dsn, user, password from %params hash (the Helios database).
This method uses the DBI->connect_cached() method to attempt to reduce the number of open connections to a particular database.
Record a message in the Helios log. If a syslogd log facility is set in helios.ini (via the syslog_facility option), the message is also logged into the specified syslogd facility.
In addition to the log message, there are two optional parameters:
The current Helios::Job object being processed. If specified, the jobid will be logged in the database along with the message.
The priority of the message as defined by syslogd. These are really integers, but if you import the syslog constants [use Sys::Syslog qw(:macros)] into your namespace, your logMsg() calls will be much more readable. Refer to the "CONSTANTS" in Sys::Syslog manpage for a list of valid syslog constants. LOG_DEBUG, LOG_INFO, LOG_WARNING, and LOG_ERR are the most common used with Helios; LOG_INFO is the default.
In addition, there are two INI file options used to configure logging. These will be passed to syslogd when logMsg() calls Sys::Syslog::openlog():
The syslog facility to log the message to.
Any logging options to specify to syslogd. Again, see the Sys::Syslog manpage.
For database logging, the host, process id, and job service class are automatically recorded in the database with your log message. If you supplied either a Job object or a priority, the jobid and/or priority will also be recorded with your log message.
Given a string of XML, parse it into a mixed hash/arrayref structure. This uses XML::Simple.
This method is DEPRECATED; its functionality has been moved to Helios::Job->parseArgXML().
Given a Helios::Job object, getJobArgs() returns a hashref representing the parsed job argument XML. It actually calls the Helios::Job object's parseArgs() method and returns its value.
These methods should be called in the subclass's run() method to mark a job as successfully completed, failed, or failed permanently. They actually call the appropriate methods of the given Helios::Job object.
Marks $job as completed successfully.
Marks $job as failed. Allows job to be retried if your subclass supports that (see max_retries()).
Marks $job as permanently failed (no more retries allowed).
Given a metajob, burstJob bursts it into its constituent jobs for other Helios workers to process. Normally Helios::Service's internal methods will take care of bursting jobs, but the method can be overridden if a job service needs special bursting capabilities.
helios.pl, Helios::Job, Helios::Error, Helios::ConfigParam, Helios::LogEntry, TheSchwartz, XML::Simple, Config::IniFiles
Andrew Johnson, <ajohnson at ittoolbox dotcom>
Copyright (C) 2008 by CEB Toolbox, Inc.
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.0 or, at your option, any later version of Perl 5 you may have available.
This software comes with no warranty of any kind.
To install Helios, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Helios
CPAN shell
perl -MCPAN -e shell install Helios
For more information on module installation, please visit the detailed CPAN module installation guide.