RCGI - Remote CGI distributed processing
use RCGI; @result = Invoke('jobone',@arguments); $result = Invoke('jobtwo',@arguments); $remote_subroutine = new RCGI($base_url,$library_path,$module,$subroutine); @my_result = $remote_subroutine->Call(@arguments); if ($remote_subroutine->Success()) { print @my_result,''; } else { print STDERR "Call to " . $remote_subroutine->Base_URL() . " failed with status: " . $remote_subroutine->Status() . ' ' . $remote_subroutine->Error_Message() . "\n"; } $remote_subroutine->Async(1); $remote_subroutine->Wantarray(1); $remote_subroutine->Call(@arguments); while(! $remote_subroutine->Done()) { # This should really be something usefull--like calls to other servers! sleep 1; } @my_result = $remote_subroutine->Read(); if ($remote_subroutine->Success()) { print @my_result,''; } else { print STDERR "Call to " . $remote_subroutine->Base_URL() . " failed with status: " . $remote_subroutine->Status() . ' ' . $remote_subroutine->Error_Message() . "\n"; } $result = RCGI::run_cgi_command($base_url, \%cgi_form, method => $method, username => $username, password => $password, timeout => $timeout, user_agent => $user_agent, nph => $bool_nph); # In a CGI script ($cgi_form, %options) = RCGI::Process_Parameters( new CGI ); $result = RCGI::run_cgi_command($base_url, $cgi_form, %options);
This perl library provides remote execution using CGI on remote web servers.
The installation of RCGI for full functionality consists of the following steps:
1) Edit RCGI/Config.pm to change the location of the configuration directory to an appropriate place.
2) Install the RCGI library itself by doing: perl Makefile.PL make make test make install
3) Put the perlcall.cgi CGI script and the SAR.pm module on every computer which will be running remote subroutines in the computer's webserver's cgi-bin directory.
4) Create sard.conf, server.conf, and jobs.conf files in the /usr/rcgi directory.
5) Start the sard daemon running on a computer which has read and write access to the /usr/rcgi directory.
6) (Optional) Edit the line in sardcheck ($sard_user = 'sard_user'), to be the user which ran the the sard daemon in step five. As the same use as step five, add a crontab entry which looks similar to:
30 * * * * /usr/local/bin/perl /path_to_sardcheck/sardcheck
Steps 2 and 3 are the only steps necessary if the load balancing calls: Invoke, Async_Invoke, or new_job RCGI will not be used. Step 3 may be neglected if only RCGI::run_cgi_command will be used.
If you wish to change the location of the configuration directory from the default value of /usr/rcgi, edit RCGI/Config.pm. The configuration must then be made, mkdir /usr/rcgi and set to the correct permissions: chgrp rcgi /usr/rcgi ; chmod g+rw /usr/rcgi. The DBM files: load.dir and load.pag are created in this directory and must be writable by any user process attempting to Invoke remote subroutines.
The sard.conf, server.conf, and jobs.conf files need to then be created in the /usr/rcgi directory. Following is the format for those files:
# machine URL_of_perlcall.cgi path_to_SAR.pm_module # Items on a line must be seperated by a single tab machine_name http://www.webserver.url/cgi-bin/perlcall.cgi path_to_SAR.pm_module
Usage is: sard /usr/rcgi/sard.conf /usr/rcgi/sar [timeperiod_in_seconds] [bool_verbose]
The sard (System Activity Report Daemon) runs in the background to collect usage from the machines configured in the sard.conf file. It uses the RCGI library to call (via perlcall.cgi) the SAR.pm module which, on Unix, uses the sar program to collect system activity over, by default, 10 minute periods. This information is stored in the DBM sar file located in the /usr/rcgi directory.
This system activity information is used by the RCGI library to implement load balancing of job requests.
# machine number_of_processors processes_per_processor reserve_idle(in percent) # the high reserve_idle should prevent those machines from being used # Items on a line must be seperated by a single tab medium 4 2 10 shared 4 1 50 dud 1 2 100000 mine 1 1 100000 super 12 1 10
# job_type server task_url library_path module subroutine option option_value # where option can be: timeout, username, password, user_agent # Items on a line must be seperated by a single tab jobone machine1 http://webserver1/cgi-bin/perlcall.cgi module_path Module subroutine option_name option_value jobone machine2 http://webserver2/cgi-bin/perlcall.cgi module_path Module subroutine option_name option_value jobtwo machine2 http://webserver2/cgi-bin/perlcall.cgi module_path Module subroutine option_name option_value jobtwo machine3 http://webserver3/cgi-bin/perlcall.cgi module_path Module subroutine option_name option_value
The perlcall.cgi perl CGI script and the SAR.pm module will need to be installed in a cgi-bin directory of the web server of every computer which will be set up to allow jobs to be Invoke'ed or Call'ed. The SAR.pm module can alternatively be installed anywhere in the standard perl @INC path.
Perl modules to call must be in the standard perl @INC path or in the library path given in the calls or in jobs.conf.
To install this package, just change to the directory in which this file is found and type the following:
perl Makefile.PL make make test make install
In order for a job to be invoked, the sard daemon must be running to collect computer processor loads. The perlcall.cgi CGI script and the SAR.pm module must be installed properly on each computer.
The RCGI library allows calling Perl subroutines in modules on remote computers with load balancing.
RCGI calculates which machine to invoke a job on by using the machine which has the maximum idle time as determined by:
1) Take the measured idle time for each machine if it is newer than the last calculated idle time for the machine.
2) Subtract the reserve_idle for each machine.
3) If two machines have similar resulting idle times, use the machine with the most increase in measured idle time.
The resulting idle time then has a process usage amount subtracted from it and which is then stored in the DBM load file stored in the /usr/rcgi directory for subsequent usage for other job requests.
The process usage for a machine is calculated according to the following formula:
process_usage = (100 / (machine_processors * processes_per_processor));
A perl program which is written as:
use lib 'module_path'; use Module; $result = Module::subroutine(@arguments); # or @result = Module::subroutine(@arguments);
can be converted to use a job, jobone:
jobs.conf entry:
jobone machine1 http://webserver1/cgi-bin/perlcall.cgi module_path Module subroutine jobone machine2 http://webserver2/cgi-bin/perlcall.cgi module_path Module subroutine
by being rewritten as:
use RCGI; $remote_subroutine = new_job RCGI('jobone'); $result = $remote_subroutine->Call(@arguments); # or @result = $remote_subroutine->Call(@arguments);
or
use RCGI; $result = Invoke('jobone',@arguments); # or @result = Invoke('jobone',@arguments);
or can be rewritten to directly call a specific machine, machine1, as:
use RCGI; $remote_subroutine = new RCGI('http://webserver1/cgi-bin/perlcall.cgi', 'module_path', 'Module', 'subroutine'); $result = $remote_subroutine->Call(@arguments); # or @result = $remote_subroutine->Call(@arguments);
with the error checking for failure of the remote call by:
if ($remote_subroutine->Success()) { print $result; } else { print "Call to " . $remote_subroutine->Base_URL() . " failed with status: " . $remote_subroutine->Status() . ' ' . $remote_subroutine->Error_Message() . "\n"; }
There are four possible uses or layers for RCGI:
1. Invoking a module subroutine as a job via perlcall.cgi on the least busy computer defined for that job type. This may be either synchronous or asynchronous. 2. Getting a RCGI remote subroutine object for the least busy computer defined for that job type. 3. Calling a module subroutine via perlcall.cgi on a particular computer. This may be either synchronous or asynchronous. 4. Retrieving HTML pages from static HTML pages or CGI scripts using RCGI::run_cgi_command.
The arrows, '# ==>', show the usable API of RCGI.
sard.conf / L sard | 1 ==> Invoke or Async_Invoke V | jobs.conf, server,conf, DBM file sar | / V L 2 ==> new_job RCGI <-----> DBM file load | | V 3 ==> new RCGI | | V $remote_subroutine->Call ==> http://www/cgi-bin/perlcall.cgi | /| A | /| | | / | V V / | return( eval ' 4 ==> RCGI::run_cgi_command / | use lib 'library_path'; / | use Module; / | Module::subroutine(@arguments); L | ' ); $result or @result V $remote_subroutine->Success $remote_subroutine->Done $remote_subroutine->Error_Message | | V $remote_subroutine->Read | | V result $remote_subroutine->Sucess $remote_subroutine->Error_Message
@my_result = Invoke('job_name',@arguments);
Invoke a job to synchronously call a remote subroutine.
Where @arguments is the normal list of arguments for the remote subroutine.
$remote_subroutine = Async_Invoke('job_name',@arguments);
Invoke a job to asynchronously call a remote subroutine.
$remote_subroutine = new_job RCGI('job_name'); OR $remote_subroutine = new_job RCGI('job_name',$minimum_load);
This will create a new object which will allow a remote subroutine call for a particular job type. $minimum_load is the minimum percentage of idle to leave when assigning jobs.
$remote_subroutine = new RCGI($base_url,$library_path,$module,$subroutine) OR $remote_subroutine = new RCGI($base_url,$library_path,$module,$subroutine, -option => value)
The arguments are:
$base_url -- the base URL for the remote subroutine call. This is the URL for perlcall.cgi on the remote web server.
$library_path -- the location of the module which contains the subroutine for the remote subroutine call. This is optional--undef may be passed instead if the module is located relative to the perl @INC path. A '.' may be passed to specify the cgi-bin directory on the remote web server.
$module -- the module which contains the subroutine for the remote subroutine call.
$subroutine -- the name of the subroutine to call for the remote subroutine call. This subroutine must be callable in the form Module_Name::subroutine();. Please remember that no executation state is maintained by default on the remote computer.
Options are passed as: -option => value, where -option is one of:
-async Do an asynchronous call. -wantarray Force array or scalar result (useful for using with async). -username Username to login to remote web server, if any. -password Password to login to remote web server, if any. -user_agent User_agent to use for remote web server. -timeout Timeout in seconds for web connection (default is 180).
This will create a new object which allows remote subroutine calls.
Synchronous Call
@my_result = $remote_subroutine->Call(@arguments); OR $my_result = $remote_subroutine->Call(@arguments);
Asynchronous Call
$remote_subroutine->Call(@arguments); while(! $remote_subroutine->Done()) { # This should really be something useful sleep 1; } @my_result = $remote_subroutine->Read(); OR $my_result = $remote_subroutine->Read();
$remote_subroutine->Done();
Return true when the asynchronous call has completed.
@my_result = $remote_subroutine->Read(); OR $my_result = $remote_subroutine->Read();
Fetch the result from an asynchronous call.
$remote_subroutine->Success()
This returns true if the remote subroutine call was completed with no errors.
$remote_subroutine->Status()
This returns the status code from the remote subroutine call. Possible values are:
-30 -- Machines are busy, the load is less than the load minimum (default is zero idle) -27 -- Missing task_url, module, or subroutine for job type definition for assigned machine -26 -- Missing job type definition for assigned machine -25 -- Missing sar measurement for assigned machine -24 -- Unable to assign machine -20 -- No job types defined match asked for job type -13 -- Unable to open load file -12 -- Unable to open sar file -11 -- Unable to open server.conf file -10 -- Unable to open jobs.conf file -1 -- The base URL, the module, or the subroutine were not given. -2 -- Unable to fork background process for asynchronous call. 200 -- Successful call. >200 -- Error code from remote web server or CGI script
$remote_subroutine->Error_Message()
This returns the associated error message, if any, from an unsuccessful remote subroutine call. If the Status is greater than 200, then the error message is from the remote web server.
$base_url = $remote_subroutine->Base_URL(); OR $remote_subroutine->Base_URL($base_url);
Get or set the base URL for the remote subroutine call. This is the URL for perlcall.cgi on the remote web server.
$library_path = $remote_subroutine->Library_Path(); OR $remote_subroutine->Library_Path($library_path);
Get or set the location of the module which contains the subroutine for the remote subroutine call. This is optional--undef may be passed instead if the module is located relative to the perl @INC path. A '.' may be passed to specify the cgi-bin directory on the remote web server.
$module = $remote_subroutine->Module(); OR $remote_subroutine->Module($module);
Get or set the module which contains the subroutine for the remote subroutine call.
$subroutine = $remote_subroutine->Subroutine(); OR $remote_subroutine->Subroutine($subroutine);
Get or set the name of the subroutine to call for the remote subroutine call. This subroutine must be callable in the form Module_Name::subroutine();. Please remember that no executation state is maintained by default on the remote computer.
$async = $remote_subroutine->Async(); OR $remote_subroutine->Async($async);
Get or set whether the call is asynchronous.
$wantarray = $remote_subroutine->Wantarray(); OR $remote_subroutine->Wantarray($wantarray);
Get or set whether the call returns a scalar or an array (or associative array).
$username = $remote_subroutine->Username(); OR $remote_subroutine->Username($username);
Get or set the username, if any, used to login to the remote server.
$password = $remote_subroutine->Password(); OR $remote_subroutine->Password($password);
Get or set the password, if any, used to login to the remote server.
$user_agent = $remote_subroutine->User_Agent(); OR $remote_subroutine->User_Agent($user_agent);
Get or set the user_agent used to when connecting to the remote server.
$timeout = $remote_subroutine->Timeout(); OR $remote_subroutine->Timeout($timeout);
Get or set the timeout in seconds used in the connection to the remote server. Default is 180 seconds.
($cgi_form, %options) = RCGI::Process_Parameters( new CGI , \%TRANSLATE, \%IGNORE );
This processes the CGI parameters, using the passed CGI object reference. The optional %TRANSLATE associative array allows passing CGI parameters with a different parameter field name (i.e., translate paramter foo=1 to bar=1). The optional %IGNORE associative array specifies CGI parameters which should not be passed on.
Returned are $cgi_form which is a reference to an associative array which contains the CGI parameters in a form ready to pass to RCGI::run_cgi_command() and the %options associative array of options to pass to RCGI::run_cgi_command().
$result = RCGI::run_cgi_command($base_url, \%cgi_form, %options);
This fetches an HTML page from either a static HTML page or a CGI script.
$base_url is the URL of the page to get. \%cgi_form is an associate array whose index is CGI parameters to pass and whose values are the CGI parameter values to pass to the remote CGI script. If a parameter's name has 'upload:' prepended to it, then the values will be passed using the multipart/form-data file upload method. (Example $cgi_form = { 'upload:seq_file' => "> sequence\nAAAAA\n" }.)
-method CGI method to use (GET is default). Values are 0 or undef for GET and 1 for POST -nph Use 1 to treat the remote CGI script as NPH. -username Username to login to remote web server, if any. -password Password to login to remote web server, if any. -user_agent User_agent to use for remote web server. -timeout Timeout in seconds for web connection (default is 180).
#!/usr/local/bin/perl use RCGI; @result = Invoke('jobtest1','one'); print @result; $result = Invoke('jobtest2','two'); print $result; exit; $remote = new_job RCGI('jobtest1'); @out = $remote->Call('one'); if ( $remote->Success() == 0) { print " Failed with error: " . $remote->Error_Message() . "\n"; undef @out; }
#!/usr/local/bin/perl # # use RCGI; $base_url = 'http://www.sandrock.edu/cgi-bin/perlcall.cgi'; $library_path = '/my/module/directory'; $module = 'MyModule'; $subroutine = 'my_subroutine'; $remote_subroutine = new RCGI($base_url,$library_path,$module,$subroutine); @my_result = $remote_subroutine->Call(0, 'a', 'b'); $, = "\n"; if ($remote_subroutine->Success()) { print @my_result,''; } else { print STDERR "Call to " . $remote_subroutine->Base_URL() . " failed with status: " . $remote_subroutine->Status() . ' ' . $remote_subroutine->Error_Message() . "\n"; } $my_result = $remote_subroutine->Call(0, 'a', 'b', 'c'); if ($remote_subroutine->Success()) { print $my_result,''; } else { print STDERR "Call to " . $remote_subroutine->Base_URL() . " failed with status: " . $remote_subroutine->Status() . ' ' . $remote_subroutine->Error_Message() . "\n"; } $remote_subroutine->Async(1); $remote_subroutine->Wantarray(1); $remote_subroutine->Call(5, 'async', 'hronous'); $| = 1; while(! $remote_subroutine->Done()) { # This should really be something usefull--like calls to other servers! sleep 1; print "."; } @my_result = $remote_subroutine->Read(); $, = "\n"; if ($remote_subroutine->Success()) { print @my_result,''; } else { print STDERR "Call to " . $remote_subroutine->Base_URL() . " failed with status: " . $remote_subroutine->Status() . ' ' . $remote_subroutine->Error_Message() . "\n"; }
When reporting bugs/problems please include as much information as possible.
A small script which yields the problem will probably be of help. If you cannot include a small script then please include a Debug trace from a run of your program which does yield the problem.
Brian H. Dunford-Shore brian@ibc.wustl.edu David J. States states@ibc.wustl.edu
Copyright 1998, Washington University School of Medicine, Institute for Biomedical Computing. All rights reserved.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
Address bug reports and comments to: www@ibc.wustl.edu
You really mean 'extra' features ;). None known.
Copyright (c) 1997 Washington University, St. Louis, Missouri. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install RCGI, copy and paste the appropriate command in to your terminal.
cpanm
cpanm RCGI
CPAN shell
perl -MCPAN -e shell install RCGI
For more information on module installation, please visit the detailed CPAN module installation guide.