helios.pl - Launch a daemon to service jobs in the Helios job processing system
export HELIOS_INI=/path/to/helios.ini [export HELIOS_DEBUG=1] helios.pl I<jobclass> [--clear-halt]
The helios.pl program, given a subclass of Helios::Worker, will launch a daemon to service Helios jobs of that subclass. The number of worker processes to run concurrently and various other parameters are set via a helios.ini file and the Helios MySQL database (the connection information of which is also defined in helios.ini).
Under normal operation, helios.pl will attempt to load the worker class specified on the command line and use it to read the contents of the helios.ini file. If successful, it will attempt to connect to the Helios parameter database and read the relevant parameters from there. If that is successful, helios.pl will daemonize and start servicing jobs of the specified class.
In debug mode (set HELIOS_DEBUG=1), helios.pl will not disconnect from the terminal, and will output extra debugging information to the screen and the logfile. It will also enable debug mode on the worker class, which may also support outputting debugging information.
If the --clear-halt is specified, helios.pl will attempt to remove a HALT parameter specified in the Helios configuration parameters (the helios_params_tb table). This is helpful if you shutdown your Helios service daemon using the Panoptes worker admin view. Note that it will NOT remove a HALT specified globally (where host = '*').
The initial parameters for helios.pl are defined in an INI-style configuration file typically named "helios.ini". The file's location is normally specified by setting the HELIOS_INI environment variable before helios.pl is started. If HELIOS_INI isn't set, helios.pl will default to the helios.ini in the current directory, if present.
A helios.ini file normally contains a [global] section with the parameters necessary to connect to the Helios parameter database, and any parameters local to the server the workers run on (e.g. log facility and options). In addition, each worker class can have a section in helios.ini containing parameters local only to them.
Example helios.ini:
[global] dsn=dbi:mysql:host=10.1.0.21;db=helios_db user=helios password=password syslog_facility=user syslog_options=nofatal,pid [ITTB::BroadcastService] master_launch_interval=60 zero_launch_interval=90 [SearchIndex::LoadTestService] OVERDRIVE=1 MAX_WORKERS=3
Datasource name for the Helios parameter database. This will also contain the data structures for the underlying TheSchwartz queuing system.
Database user for the datasource name described above.
Database password for the datasource name described above.
The syslog log facility to send log messages. The worker class will also log messages to this facility. If a syslog_facility isn't defined, logging to syslog is disabled. (Uses Sys::Syslog CPAN module.)
Any desired options for logging to the syslog facility.
The location where helios.pl should write the PID file for this worker. The default is "/var/run/helios". The name of the PID file will be a variation on the worker class's name.
Set the master_launch_interval to determine how long helios.pl should sleep after launching jobs before accessing the database to update its configuration parameters and check for waiting jobs. The default is 1 second which works well for normally short-lived jobs (2 secs or less), but that may be overkill for longer-lived jobs (jobs that run longer than 2 sec). Setting the master_launch_interval option helps prevent needless database traffic.
Set the zero_launch_interval to determine how long helios.pl should sleep after reaching its MAX_WORKERS limit. The default is 2 sec. If jobs are running long enough that helios.pl is frequently hitting is MAX_WORKERS limit (there are waiting jobs but helios.pl can't launch new workers because previously launched jobs are still running), increasing the zero_launch_interval will reduce needless database traffic.
In addition to helios.ini, certain helios.pl configuration options can be set via the Ctrl Panel in the Helios Panoptes web interface. These configuration options are read by helios.pl from the helios_params_tb table in the Helios database. Though the Ctrl Panel is designed mostly to provide a standardized interface for worker classes to store and retrieve configuration parameters, certain helios.pl control parameters can be read from here as well. The helios.pl worker daemon will read the parameters via the worker class's getParams() method, so parameters can be set for a specific class on a specific host.
The helios.pl daemon will refresh its configuration parameters from the database after waiting master_launch_interval seconds after launching worker child processes. If MAX_WORKERS workers are still running, the helios.pl daemon will wait zero_launch_interval seconds to refresh its configuration parameters and try to launch workers again. If you have long running jobs (jobs that last more than a few seconds), resetting these parameters higher may help smooth processing and reduce needless database traffic. See the HELIOS OPERATION section below for more information.
The Helios system is designed to run one daemon process for each worker class per server. However, each daemon can run more than 1 worker child process simultaneously. The upper limit for worker processes is set using the MAX_WORKERS option. The helios.pl default is 1 worker per server. You can provide a different default for a particular class by specifying a value in helios.ini. If active management of available workers is not necessary for a particular worker class, this option can be set in helios.ini.
Speed up job processing by allowing workers to service jobs until no more are in the queue. Normally, a worker process will service a single job and then exit (1:1 worker/job ratio). With OVERDRIVE mode enabled, a worker process will keep servicing jobs until it can find no more of that type in the job queue (1:many worker/job ratio). This may help to speed up processing for short-lived jobs, as once worker child processes have started, they stay in memory, continuing to process jobs until there are no more to process. This can also enable on-the-fly caching of resources, as once a worker has loaded a resource or established a database connection for a job it can be held for use by subsequent jobs, potentially reducing database load or disk I/O. Longer running jobs may not benefit as much, however.
A few extra precautions should be taken in a worker class in order to support OVERDRIVE mode. If the worker class should explicitly exit() if it encounters an error to prevent the error from affecting subsequent jobs. After processing a job successfully (at the end of the work() method) the process should also exit if the Helios::Worker->shouldExitOverdrive() returns a true value.
Like MAX_WORKERS, if active management of the job processing mode is not necessary, this parameter can be set in helios.ini (though that is strongly discouraged).
If set to 1, a HOLD parameter will cause the helios.pl worker daemon to stop launching new workers to process jobs. Currently processing jobs will be allowed to complete, and those worker child processes will end normally. HOLD = 1 will also cause the Helios::Worker->shouldExitOverdrive() method to return a true value, which should cause workers that correctly support that mode to end after their current job is completed.
Setting HOLD to 0 will allow normal job processing to resume.
If a HALT parameter is defined, the helios.pl worker daemon will start process cleanup and will exit. As with HOLD = 1, jobs currently processing will complete, and the worker child processes will exit normally. HALT will cause Helios::Worker->shouldExitOverdrive() to return a true value, which should cause workers supporting OVERDRIVE mode to exit after they have finished the current job.
Once a helios.pl daemon has HALTed, there is no way to restart it from the Helios Pantoptes interface. It must be restarted from the command line with a 'helios.pl <worker::class>' command.
After initial setup, the helios.pl daemon will enter a main operation loop where configuration parameters are refreshed, the job queue is checked, and worker child processes are launched and cleaned up after. A HOLD = 1 parameter will temporarily cause the loop to pause processing, while a HALT parameter will cause the helios.pl daemon to exit the loop, clean up, and exit.
There are several steps in the helios.pl main operation loop:
Refresh configuration parameters from database
Check job queue to see if there are jobs available for processing (if not, sleep for zero_sleep_interval seconds and start again).
If there are jobs available, check to see how many worker child processes are currently running. If MAX_WORKERS workers are already running, sleep for zero_launch_interval seconds and start again. The zero_launch_interval setting should be long enough to allow at least some of the running jobs to complete.
Subtract the number of currently running workers from MAX_WORKERS and launch that many workers to handle the available jobs. If MAX_WORKERS is 5 and only 2 workers are running, that means the helios.pl daemon will launch 3 workers.
Sleep master_launch_interval seconds and start again. The master_launch_interval setting should be long enough to allow at least some of the jobs to complete.
The launch_worker() function launches a new TheSchwartz worker of the type specified on the command line. After the fork() from the main process, if the process is the child, it calls launch_worker() to instantiate a new TheSchwartz object, identify the object as one of a certain job type (TheSchwartz calls it ability), and starts the worker on its way by calling the work() (or work_until_done(), if OVERDRIVE is enabled) method.
The reaper() function is responsible for cleaning up after dead child processes. It's called when helios.pl receives a SIG_CHLD signal. The function reaps any children with waitpid(), removes the child's PID from the $workers hash of running workers, and re-establishes itself as the signal handler for the next SIG_CHLD signal.
Don't fear the reaper.
The daemonize() function is called to turn the Helios program into a daemon servicing jobs of a particular class. It forks a new process, which disconnects from the launching terminal.
Normally, daemonization also including setting up signal handling, but daemonize() isn't called in debug mode, so signal traps are actually set up in the main program.
Writes a PID file to a location (defaults to /var/run/helios) to track which daemons are running. The file will be named after the worker class running, all lowercase, with colons replaced by underscores. For example, the PID file for a worker class named 'SearchIndex::LoadTestWorker' will be named "searchindex__loadtestworker.pid". To change the location where the PID file is created, set the pid_path option in helios.ini.
During a clean shutdown, the PID file should be removed. If the daemon encountered an unrecoverable error, this function shouldn't be called, and a cron job on the server should notice the process has disappeared and restart it.
Given the pid_path, check to see if a $pid_file for the loaded service class exists and, if it does, check to see if that process is still running. If the file doesn't exist or it does but isn't running, this function returns 0. If the process is still running, record the error and return the running process's pid to signal that service startup should halt.
If the --clear-halt option is specified on the command line, clear_halt() is called to attempt to clear the HALT parameter in the helios_params_tb. For safety reasons, it only clears a HALT for the loaded service class AND the specific host helios.pl is running on; it will not clear a global HALT parameter (where the host is specified as '*').
The clean_shutdown function is called when helios.pl is intentionally shutdown (setting a parameter of HALT with a value of 1 in helios_params_tb). It removes the PID file created on startup and unregisters the worker daemon from the database.
The register() function records information about the currently running worker daemon in the database. The register() function is designed to be run every $REGISTRATION_INTERVAL seconds. That way, if a worker daemon dies off unexpectedly (without calling unregister()), it can be determined that something has happened to the daemon and it possibly needs to be restarted.
(In reality, register() and unregister() are only necessary to provide a display for Panoptes, to more easily assess system status and facilitate the HALTing of worker daemons or HOLDing of jobs.)
The unregister() function removes any record of the currently running daemon from the database. It should be called whenever there is a clean shutdown.
Helios, Helios::Service
Andrew Johnson, <ajohnson@ittoolbox.com>
Copyright (C) 2007-8 by CEB Toolbox, Inc.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.0 or, at your option, any later version of Perl 5 you may have available.
This software comes with no warranty of any kind.
To install Helios, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Helios
CPAN shell
perl -MCPAN -e shell install Helios
For more information on module installation, please visit the detailed CPAN module installation guide.