helios.pl - Launch a daemon to service jobs in the Helios job processing system
# Make sure HELIOS_INI is set and exported. Optionally enable debug mode. export HELIOS_INI=/path/to/helios.ini [export HELIOS_DEBUG=1] # Full command line options helios.pl [--service=<service class>] [--jobtypes=<jobtypename,jobtypename>] \ [--clear-halt] [--version] [--help] # Simple cmd line example: start a daemon with the Helios::TestService service. helios.pl Helios::TestService # More complex: start a Helios::TestService daemon, but have it run MyService jobs. helios.pl --service=Helios::TestService --jobtypes=MyService # Just prints version info. helios.pl --version # Prints this help page. helios.pl --help
The helios.pl program, given a Helios service class, will launch a daemon to service Helios jobs of that class. The number of worker processes to run concurrently and various other parameters are set via a helios.ini file and the Helios collective database (the connection information of which is also defined in helios.ini).
Under normal operation, helios.pl will attempt to load the service class specified on the command line and read the contents of the helios.ini file. If successful, it will attempt to connect to the Helios collective database specified in helios.ini and read the configuration parameters relevant to the loaded service class from there. If that is successful, helios.pl will then daemonize and start servicing jobs of the specified class. If additional jobtypes are specified with the --jobtypes option, jobs of those additional types will also be serviced by the loaded service class.
The --service option specifies the name of the Helios service to load and launch worker processes for. If you specify the name as the first option on the command line, you can actually drop the '--service=' part, as helios.pl will assume the first option is the service class name.
Examples:
helios.pl --service=MyService --OR-- helios.pl MyService
Normally, a Helios service will run jobs belonging to its own jobtype (the service class name matches the jobtype name). However, as of Helios 2.80, a Helios service can run jobs of multiple jobtypes if necessary. These additional jobtypes should be specified on the helios.pl command line with the --jobtypes option. You can specify multiple jobtypes by separating them with commas, or specify multiple --jobtype values. (Thanks to helios.pl's use of Getopt::Long, '--jobtype' and '--jobtypes' are interchangable.)
# MyService handles jobs of jobtype "MyService" (the default) helios.pl MyService # MyService handles MyService jobs and MyIndexer jobs helios.pl MyService --jobtype=MyIndexer # MyService handles MyService, MyIndexer, and MyMailer jobs helios.pl MyService --jobtypes=MyIndexer,MyMailer --OR-- helios.pl MyService --jobtype=MyIndexer --jobtype=MyMailer
For more information about jobtypes and how they relate to jobs and services, see the Helios::JobType man page.
If the --clear-halt option is specified, helios.pl will attempt to remove a HALT parameter specified in the Helios configuration for the specified service on the current host. This is helpful if you shutdown your Helios service daemon using the Helios::Panoptes Collective Admin view. Note that it will NOT remove a HALT specified globally (where host = '*').
The --version option displays the Helios framework version and the helios.pl version and then exits.
The --help option displays this documentation.
The Helios framework relies directly on two environment variables to function correctly. Both should be exported as well as set to ensure the helios.pl child processes can see them as well as the main helios.pl daemon.
The full path to the helios.ini configuration file. This should be an absolute path, not a relative one. The default is to use a helios.ini in the current directory if it exists, which will work if you are running in Debug Mode (see HELIOS_DEBUG below), but will cause problems if helios.pl is running normally.
Default: ./helios.ini
Setting HELIOS_DEBUG to 1 causes helios.pl to run in Debug Mode. In Debug Mode, helios.pl will not disconnect from the terminal and will output extra debugging information to the screen and the Helios log. It will also enable debug mode on the service class, which may support the output of extra debugging information if you so choose.
Default: undef
The initial parameters for helios.pl are defined in an INI-style configuration file typically named "helios.ini". The file's location is normally specified by setting the HELIOS_INI environment variable before helios.pl is started.
A helios.ini file normally contains a [global] section with the parameters necessary to connect to the Helios collective database, and any parameters local to the host on which the Helios service is currently running. In addition, each service class may have its own section in helios.ini containing parameters specfic to that service on that host.
Example helios.ini:
[global] dsn=dbi:mysql:host=10.1.0.21;db=helios_db user=helios password=password [LongRunningJobService] master_launch_interval=60 zero_launch_interval=90 [AnotherService] OVERDRIVE=1 MAX_WORKERS=3
This section just covers the basic config options for helios.pl. For a complete list of built-in Helios config options, consult the Helios::Configuration man page.
These options will be the same for all Helios services running a host that share the same collective database, so these options must be placed in a helios.ini section named [global] so they are visible to all running helios.pl instances.
Datasource name for the Helios collective database. The collective database houses the data structures for service configuration, jobtypes, job queuing and history, and the default logging subsystem.
Default: none
Database user for the datasource name described above.
Database password for the datasource name described above.
The location where helios.pl should write the PID file for this service instance. The name of the PID file will be a variation on the service class's name.
Default: /var/run/helios
The options listed in this section are available to tune helios.pl to work better with your Helios service class and the jobs it needs to service. These are read at startup and, unlike some other Helios config options, cannot be dynamically changed while helios.pl is running. If you wish to tune one of these parameters, reset the parameter and restart the service daemon.
Set the master_launch_interval to determine how long helios.pl should sleep after launching workers before accessing the database to update its configuration parameters and check for waiting jobs. The default is 1 second, which should be sufficient for most applications.
Default: 1
Set the zero_launch_interval to determine how long helios.pl should sleep after reaching its MAX_WORKERS limit. The default is 10 sec. If jobs are running long enough that helios.pl is frequently hitting the MAX_WORKERS limit (there are waiting jobs but helios.pl can't launch new workers because previously launched jobs are still running), increasing the zero_launch_interval will reduce needless database traffic.
Default: 10
Set the zero_sleep_interval to adjust the amount of time between checks for available jobs in the job queue when the job queue is empty. If the helios.pl daemon determines there are no available jobs for a service, it sleeps zero_sleep_interval seconds and then checks for jobs again. The default is 10 sec. If you notice jobs are sitting in the job queue too long before workers are launched to service them, reduce this number to cause jobs to be started faster. If you have a small number of jobs and do not care if they sit in the job queue for a few seconds before being serviced, increase this number to reduce database queries.
In addition to helios.ini, certain helios.pl configuration options can be set via the helios_config_* commands or the Ctrl Panel or Collective Admin views in the Helios::Panoptes web admin interface. These configuration options are read by helios.pl from the HELIOS_PARAMS_TB table in the Helios collective database. For more information on configuration parameters available to tune how a service runs jobs, see the Helios::Configuration man page. For more information on the helios_config_get, helios_config_set, helios_config_unset commands and the Helios::Panoptes web admin interface, see their respective man pages or Perl POD.
After initial setup, the helios.pl daemon will enter a main operation loop where configuration parameters are refreshed, the job queue is checked, and worker processes are launched and cleaned up after. A HOLD = 1 parameter will temporarily cause the loop to pause processing, while a HALT parameter will cause the helios.pl daemon to exit the loop, clean up, and exit.
There are several steps in the helios.pl main operation loop:
Refresh configuration parameters from database.
Check job queue to see if there are jobs of the correct jobtype(s) available for processing (if not, sleep for zero_sleep_interval seconds and start again).
If there are jobs available, check to see how many worker processes are currently running. If MAX_WORKERS workers are already running, sleep for zero_launch_interval seconds and start again. The zero_launch_interval setting should be long enough to allow at least some of the running jobs to complete (the default is 10 secs).
If there are jobs available and the MAX_WORKERS limit has not been reached, determine the number of additional workers to launch and launch them. The number of workers to launch is governed by the WORKER_LAUNCH_PATTERN configuration parameter; see the Helios::Configuration man page for more information.
Sleep master_launch_interval seconds and start the operation loop again.
The helios.pl program calls several of its own functions at various times during startup, shutdown, and during main loop operation. These are probably only of limited interest, but are documented here for those wanting to understand better the guts of the helios.pl daemon.
The daemonize() function is called to turn the Helios program into a daemon servicing jobs of a particular class. It forks a new process, which disconnects from the launching terminal.
Normally, daemonization would also include setting up signal handling, but the daemonize() function isn't called in debug mode, so signal traps are actually set up in the main program.
The double_clutch() function implements the WORKER_MAX_TTL functionality. If the WORKER_MAX_TTL parameter is set for a service, the service daemon periodically calls double_clutch() to check the workers and clean up any that have run too long. The double_clutch() function waits a certain amount of time (WORKER_MAX_TTL_WAIT_INTERVAL secs), then checks the amount of time each of the running workers has been active. If a worker has been running longer than the service's WORKER_MAX_TTL, the worker is killed (by sending it a SIGKILL signal).
The launch_worker() function launches a new worker process. After the fork() from the main process, the new child process will call launch_worker() to instantiate and configure a new Helios::TS job queuing system object and start the worker on its way by calling the object's work() method. If the OVERDRIVE run mode is enabled for the service, the work_until_done() method is called instead. See Helios::Configuration for information about running workers in OVERDRIVE vs. Normal Mode.
The reaper() function is responsible for cleaning up after dead child processes. It's called when helios.pl receives a SIG_CHLD signal. The function reaps any children with waitpid(), removes the children's PID from the $workers hash of running workers, and re-establishes itself as the signal handler for the next SIG_CHLD signal.
The terminator() function is responsible for shutting down a Helios service instance when helios.pl receives a SIGTERM signal. The shutdown process performs several steps:
Sets HALT in the Helios config params for the loaded service, so the worker processes know to exit at the end of their current job
Sleeps a certain amount of time (zero_launch_interval x 2 secs) to allow worker processes to complete the jobs they are processing and exit
Reaps any ended worker processes, and forcably kills any that are still running. After zero_launch_interval seconds, workers for a particular service should have exited if HALT is set. If you use this method of shutting down Helios services and repeatedly see a particular service's workers not exiting properly, increase the value of the zero_launch_interval configuration parameter for that service in your helios.ini or Helios::Panoptes Ctrl Panel.
Clears the HALT parameter set earlier, since all of the workers are now shut down (one way or the other).
Removes the service's PID file written in the pid_path set in helios.ini
Unregister's with the Helios collective's registry table in the Helios database
Issues a final log message indicating the loaded service has shut down on the current host
Once all these steps are complete, the helios.pl program exits.
Writes a PID file to a location (defaults to /var/run/helios) to track which daemons are running. The file will be named after the service class running, all lowercase, with colons replaced by underscores. For example, the PID file for a service class named 'SearchIndex::LoadTestWorker' will be named "searchindex__loadtestworker.pid". To change the location where the PID file is created, set the pid_path option in helios.ini.
The running_process_check() function should be run before this function to exclusively lock the PID file.
During a clean shutdown, remove_pid_file() is called to delete the PID file associated with the service daemon.
Given the pid_path, check to see if a $pid_file for the loaded service class exists and, if it does, check to see if that process is still running. If the file doesn't exist or it does but isn't running, this function returns 0. If the process is still running, record the error and return the running process's pid to signal that service startup should halt.
If the PID file does not exist, running_process_check() creates it. The PID file is then exclusively locked to prevent other daemons for the same service from writing to it. This helps ensure multiple Helios daemons for same service will not start up at the same time.
The clean_shutdown function is called when helios.pl is intentionally shutdown (setting a HALT parameter in the Helios::Panoptes Ctrl Panel or sending the helios.pl process a TERM signal). It removes the PID file created on startup and unregisters the service instance from the collective database.
The register() function records information about the currently running worker daemon in the database. The register() function is designed to be run every $REGISTRATION_INTERVAL seconds. That way, if a service daemon dies off unexpectedly (without calling unregister()), it can be determined that something has happened to the daemon and it possibly needs to be restarted.
(In reality, register() and unregister() are only necessary to provide a display for Panoptes, to more easily assess system status and facilitate the HALTing of service daemons or HOLDing of job processing.)
The unregister() function removes any record of the currently running daemon from the database. It is called whenever there is a clean shutdown.
Set a HALT parameter for the currently loaded service on the current host to signal to all of the worker processes that they need to exit. This function is used by the terminator() function to safely signal to workers they need to exit when the current job is completed.
If the --clear-halt option is specified on the command line, clear_halt() is called to attempt to clear the HALT parameter in the HELIOS_PARAMS_TB. For safety reasons, it only clears a HALT for the loaded service class AND the specific host helios.pl is running on; it will not clear a global HALT parameter (where the host is specified as '*').
Given the number of waiting jobs, currently running workers, and maximum workers to launch, workers_to_launch() returns the number of worker processes helios.pl should launch to correctly handle the current job workload. There are several different algorithms helios.pl can use to determine this number, which is dependent on the environment, available resources, and type of application. See Helios::Configuration for the list of possible values and an explanation of the worker launching algorithms.
Helios, Helios::Tutorial, Helios::Service, Helios::Configuration
Andrew Johnson, <lajandy at cpan dotorg>
Copyright (C) 2007-9 by CEB Toolbox, Inc.
Portions of this software, where noted, are Copyright (C) 2012-2014 by Logical Helion, LLC.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.0 or, at your option, any later version of Perl 5 you may have available.
This software comes with no warranty of any kind.
To install Helios, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Helios
CPAN shell
perl -MCPAN -e shell install Helios
For more information on module installation, please visit the detailed CPAN module installation guide.