The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.
 / 
Dicop-Server-3.03
++
/ Dicop::Data

NAME

Dicop::Data -- contains all the jobs (with chunks), proxies, testcases, clients etc

SYNOPSIS

        use Dicop::Data;

        $data = Dicop::Data->new();

        $job = $data->get_job(5);       # return the job object #5
        $job->get_chunk(3);             # get the chunk w/ id 3 from this job

See dicopd and dicopp on how to use this.

REQUIRES

perl5.005, Dicop::Base, Dicop::Item, Dicop::Data::Charset, Dicop::Data::Job, Dicop::Data::Charset::Dictionary, Dicop::Data::Client, Dicop::Data::Proxy, Dicop::Request, Dicop::Data::Result, Dicop::Data::Chunk, Dicop::Data::Group, Dicop::Data::User, Dicop::Data::Jobtype, Dicop::Data::Case, Dicop::Data::Testcase, Dicop::Item, Dicop::Config, Dicop::Client, Dicop::Security, Mail::Sendmail, Dicop::Event, Time::HiRes, Dicop::Files, File::Spec

EXPORTS

Exports nothing.

DESCRIPTION

Upon creating such an object, the server locks a file and reads it's data into memory. Upon destroying the object, the lock is released and possible changes are written back to the disk. From time to time the modified data is written back to the disk.

For each client-connect an extra file is locked to prevent from multiply client-connects to interfere with each other. This lock is released after the response was sent to the client.

All the data is read immidiately, but only written back if changed or a certain time period has passed.

METHODS

output()

result_list()

get_job, get_proxy, get_charset, get_jobtype, get_testcase, get_result, get_client, get_group

Given an ID, return the appropriate object.

        print $self->get_proxy(1);              # get proxy w/ id 1

jobs/proxies/charsets/jobtypes/testcases/results/groups/clients

Return the appropiate number of objects.

        print $self->proxies();         # how many proxies do I know

get_object

        $object = $data->get_object( { type => 'proxy', id => 123 }, $noerror );

General case of the get_foo() methods. Does not throw an error if $noerror is true, this can be used to check for the existance of an object with a certain type and ID.

get_highest_priority()

Find the job with highest priority and return it.

adjust_job_priorities()

For all running jobs calculate their job priority and store them, returns a job that matches a minimum priority.

del_user(), del_charset(), del_job(), del_client(), del_group()

Check if we can delete this object, and if so, remove any references to the object in question. Returns an error message if the deletion is not possible, otherwise undef.

cfg_default()

Given a set of keys and their values (a list or hash reference), sets these values as default in the internal cfg object, unless the key is already defined there.

check()

Applies self-check on startup and crumbles if errors in data structure are present.

parse_requests()

This parses the form parameters as send by the client (via GET/PUT) and breaks them into requests. It then sorts the requests into groups and returns references to these groups (as arrays):

        ($auth,$info,$reports,$requests,$forms) = $self->parse_requests();

handle_requests()

Takes the returned request groups from parse_requests and handles them after some basic checks, like for maximum number of requests, existing authentication etc.

This also prints the result back to the client on STDOUT.

hash_file()

        $self->hash_file($file, $optional_hash_name);

Find a file, hash it and return a message for the client to download this file. Upon error, returns the proper error response to the client.

Uses hash() internally.

hash()

        $self->hash($file);
        $self->hash($file,'target');
        $self->hash($file,'dict');

        $self->hash($filename,'dict', \"file data");

Make a hash from a file and return it in compact form as something like 123456789abcdef. Upon errors like the file not being found or unreadable, a reference to the error message is returned.

The second optional parameter is the name of the hashkey were to put the file hash.

The three-arg form uses the data stored in the third argument as the actual file contents without reading the file from disk. The filename is still used to store the hash under a unique key.

worker_hash()

Construct the worker name from config's worker_dir, arch (as sent by client), and the requested jobtype. It then makes or updates the hash for that worker executable and returns it in compact form as 'hash_123456789abcdef'.

Upon error, it returns a ref to an error message.

This is used by both request_work and request_test.

extra_files()

        my $response = $object->extra_files($job, $arch);

Returns a string containing a respone for the client with all the extra files needed to work on the given job on the given architecture.

log_msg()

Return a message string by number, along with embedded parameters. Works just like Event::msg, but it also logs the message to a logfile, depending on log_level and the message code.

Typical usage:

        $self->log_msg(430,$type,'id',$id);

_create_event()

        $self->_create_event( $event_name, $job);
        $self->_create_event( $event_name, $job, $result);

Examples:

        $self->_create_event( 'job_failed', $job);

email()

Prepare an email with an announcement by reading in a template text, completing it with the actual informations and then putting the email into the send queue.

Typical call:

        $self->email($type,$cc, $job,$chunk,$client,$result, $rc);

job, chunk, $<client> and result can be undefined and will be skipped then (they do not make sense for every email template). type is the name of the actual mail template without path and extension. cc contains the address(es) to Cc: on the mail.

chunk_list()

Create a table of all chunks of one job from an HTML template. See also check_list().

check_list()

Create a table of all the chunks in the checklist of one job from an HTML template. See also chunk_list().

job_list()

Create a table of all the jobs belonging to a certain case from an HTML template. See also chunk_list().

sorted_job_list

Used by status_main() and job_list() (indirectly via status_case()) to generate a sorted and filtered list of jobs.

check_clients()

This checks all clients for whether they are still online (aka returning results) or went offline. For each client no longer online, an email will be sent to the administrator using appropriate template (e.g. "offline.txt").

Will return a hash which keys are the id's of the clients that went offline. That is used by the testsuite.

reference_speed()

Calculate the entire cluster speed base on the reference client. Returns a integer number that represents the count of "reference clients" the cluster has.

speed()

Calculates the speed of the cluster for a particulary job in keys/s. In list context returns speed and average speed per client.

This considers only the clients that were active in the given timeframe. Will also skip clients that failed to often, or never worked on a chunk before. The returned speed is corrected by the percentage of the job, e.g a job that currently 10% of the cluster speed will return only 1/10 of it's current speed value.

        $self->speed($job->id(),3600*6);        # consider last 6 hours

report()

Client want's to report work or test case result, so check the result in.

request_file()

The client requested us to tell him one or more URIs for a particular file.

request_work()

Client requested work from us, so try to find a suitable chunk and give it to him.

request_test()

Client requested test cases (or server determined it was time to send the tests to the client again), so send him all or only the testcases for a specifiy jobtype.

help()

Create as HTML output either a help overview page (type eq 'list') or a help page to a certain topic (type is the topic name).

status_chunks()

Create as HTML output a table of all open chunks across all jobs.

status_clients()

Create as HTML output a status page about all the clients, sorted by their speed, name, keys done so far, id or online status.

status_casebyname()

Generate HTML statistics about a case specified by the name of the case (opposed to its ID).

status_client()

Generate HTML statistics about a specific client.

status_jobresults

Generate an HTML page with the results for the specified job only.

status_debug()

Generate HTML statistics with debug output.

terminate_clients

        $self->terminate_clients();

Flags all clients so that upon next connect they will terminate immidiately and can be restarted by the outer client script, effectively forcing an upgrade of all clients.

terminate_client()

Flags this client so that upon the next connect it will terminate immidiately. See also terminate_client() .

reset()

        $self->reset($request);

Returns a template (reset.txt) telling the user that the reset of an item was successfully. Called by reset_client.

reset_client()

        $self->reset_client($request);

Resets a client and then calls $self-reset()> to return a template.

reset_clients()

        $self->reset_clients($request);

Resets all clients and then calls $self-reset()> to return a template.

status_clientmap()

Create as HTML output a table showing a small colored field for each client, denoting their online or offline status.

status_proxies()

Create as HTML output a status page about all the proxies, sorted by their speed, name, keys done so far, id or online status.

status_main()

Create as HTML output a status page about one the main server status, e.g. the list of running (or other combinations, like suspended + done) jobs.

_flush_data()

This writes the data back to disk.

_clear_email_queue()

        $self->_clear_email_queue();

Delete everything in the email send queue. Mainly used by the testsuite.

flush_email_queue()

Tries to send all mails in the queue, and return the number of mails successfully sent. This is called outside the code that handles a client request, so that the client does not need to wait until all the emails are sent.

This routine will also put all to-be-sent emails into a logfile.

_search()

Searches the database and then returns a list of all matches.

get_random_job()

Get one of the running jobs at random. Returns undef if no jobs are in the running state.

write_charsets_def()

See Dicop::Files.

description_file()

See Dicop::Files.

type()

        $data->type();

Return the type of the server as string, e.g. either 'server' or 'proxy'.

search()

Generate the list of search results via _search() and return an HTML page with these results.

status_config()

Generate an HTML page showing all the current configuration settings.

status_server()

Generates the "detailed status" page and contains statistics about our status, the running time, connects/request counters etc.

read_template()

Read a 'normal' template file from the template dir, honouring styles (e.g. the latter override the general templates). Inside the template text, includes file via ##include_filename.inc##.

Returns the 'finished' template, ready to be filled with data.

read_table_template()

Works just like read_template(), except that it looks for an embedded table template via <!-- start --> and <!-- end --> and generates both the normal template text and a template for one row of the embedded table.

This also honours styles and includes files vi ##include_filename.inc##.

add_result()

Add a result to our result-list.

add_item()

Add an item to server's in-memory database.

add_clients()

        $self->add_clients ($request, $client, $info_request);

Add a range of clients to the server. Called by add(), and then itself calls add() in turn for each client to be added.

add()

        $self->add ($request, $client, $info_request, $no_template);

Creates and adds one item to the in-memory database. Returns a template filles with the fields of the newly created item. Returns an error if the item could not be constructed or added.

If the last parameter no_template is defined, no template will be loaded from disk, instead the return value will be the ID of the added object.

insert_object()

Insert an object with all it's parameters into a HTML template, e.g. fills the template with the fields from the object.

confirm()

Generate an HTML form that asks for confirmation of deletion of some object.

_include_template()

Find ##include_filename.inc## inside a template and include the file there.

read_dictionaries()

Creates a list of all available dictionary files so that they can be inserted into the HTML form for adding a dictionary charset. Returns a hash reference mapping filenames to their descriptions.

check_auth_request()

Check the auth or info request a client/proxy sent us for basic validity. Returns either a ref to the client/proxy referenced by the auth request, or an error message.

del()

Really deletes an object from the database after the confirmation was sent in.

del_item()

Check whether we can really delete an item from the server's in-memory database. If not, return error message, otherwise delete item.

status_charset()

Create a HTML status page with the details of a specific character set.

BUGS

See the BUGS file for details.

AUTHOR

(c) Bundesamt fuer Sicherheit in der Informationstechnik 1998-2006

DiCoP is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License version 2 as published by the Free Software Foundation.

See http://www.bsi.de/ for more information.