The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.
 / 
Dicop-Server-3.03
++
/ NEW

NAME

NEW -- What's new

CHANGES in v3.00

This document details the changes between the last and the current version.

REFACTORED CODE

Although on the outside, v3.00 looks almost exactly like v2.23, this release is quite different under the hood. The code was split into generic and DiCoP specific parts and all the generic parts were moved to the package Dicop-Base. This allows us to reuse the generic parts for other projects.

Here are some of the under-the-hood changes that make life for us (the developers) easier, but shouldn't affect you directly:

Request patterns

The requests (the messages exchanged between client or browser and the server) are no longer hardcoded into the source. Instead a text file def/request.def defines now the allowed requests with their parameters and options. There should be no need to edit this file, though.

Object patterns

Likewise, the internal server objects (like jobs, testcases and so on) are now defined in def/objects.def. There should be no need to edit this file, either.

Better template system

Edit and add forms are now generated via a general edit_object.tpl respectively add_object.tpl template, which automatically includes the relevant edit fields from small include files called editfield_FOO.inc. This makes maintaining and/or changing them much easier and prevents mistakes from Copy&Paste.

Here is a list of some of the changes that might affect you:

DESIGN CHANGES

Test::Simple v0.47 and Perl v5.8.3 requirements

We started to move over the testsuite to Test::Simple - this means this module is now required to run the testsuite. However, since Dicop::Server now needs at least Perl v5.8.3 to run, you will likely already have the required Test::Simple module.

IO::Socket::SLL (and OpenSSL and Net::SSLeay) requirements

If you plan to use the new SSL feature of the server, or client, then these modules must be installed.

The client requirements

The client needs quite a few parts of Dicop::Base. However, we seperated the things so that you can simple drop a few of the Dicop::Base .pm files into the client dir, and have it work without making it nec. to install Mail::Sendmail, Net::Server etc at the node.

Here is a short list of files the client needs:

        lib/basics
        lib/Dicop.pm
        lib/Dicop/Base.pm
        lib/Dicop/Cache.pm
        lib/Dicop/Client.pm
        lib/Dicop/Config.pm
        lib/Dicop/Event.pm
        lib/Dicop/Hash.pm
        lib/Dicop/Item.pm
        lib/Dicop/Request.pm

        lib/Dicop/Client/LWP.pm
        lib/Dicop/Client/wget.pm

        lib/Dicop/Request/Pattern.pm

        lib/Linux/Cpuinfo.pm

The other solution is to install Dicop::Base and all it's prerequisites at every node.

To make it easier to deploy clients we publish a Dicop-Client-3.00.tar.gz package at our website, which contains everything the client needs, except libwww and Linux::Cpuinfo, which can be found on http://search.cpan.org.

WEB INTERFACE (GUI)

The web-frontend (e.g the GUI you see when you connect to the server with a browser) has got many small fixes and enhancements. This includes:

  • graphical percentage bars instead of just plain text

  • mouse-over titles for many links and texts that give you more information without clicking or burdening the view with lots of (usually not needed) text

  • menus in the upper-right corner for Help/Edit/Delete etc instead of hard to find text-links buried in the middle of the page

  • 'Browse' buttons allow you to select files/directories from a list of existing files instead of manually typing their path and name into an edit field

  • all edit and add forms are now automatically puzzled together from templates for each valid field of the object, instead of generated from a single template file per object type. This allows much easier adjusting of these formulars and makes a lot of improvements like grouping (both by indending and coloring) possible.

BUG FIXES

Here is a listing of important bugs that have been fixed, testcases were added to prevent these from happening again:

  • Embedding the timestamp in the mail-templates results now in a readable time, instead of just the seconds since 1970. Not everybody might be able to do the conversion in his head ;)

  • Editing the minlen of a Jobtype is now possible.

  • Editing the settings of a proxy is now again possible.

  • It is now possible to give a file as a target for a testcase (as it always was possible for normal jobs), and if there was a script associated with the testcase's jobtype, then this script will be run and generate a .tgt file for the testcase.

  • Setting a field of an object to the empty string was not possible since all empty browser params were filtered away.

    Formerly this wasn't actually necessary, but now cases can have empty URL fields.

  • Errors in a config file now cause the server to actually report a proper warning including filename and line number where they are encountered.

  • Obsolete keys in the config are now detected, and produce an error upon startup.

  • The client now first gathers all filenames it needs to do it's work, then checks which ones are missing/outdated and then asks the server in one connect to get the download locations of the missing/outdated files. It then downloads them one by one, as usual.

    This solves the problem that fresh clients that need to download a lot of workers (f.i. just to work on the testcases) caused a lot of connects to the server to ask for the download locations of the files. And with lots of connects in a short time frame the client could hit the rate-limit even before it did any real work, since requests for filenames count towards that limit.

    This also reduces the number of connects to the server, and hence it's load.

  • When adding simple charsets, multiple sequences are now handled correctly. This means that you can enter things like 'a'..'z', '0'..'9' and it will work as expected. Formerly only one sequence was allowed, contradicting what the help said.

  • Checking other jobs with a found result did not work properly, and the check list (e.g. the list containing the results from other jobs with the same charset) did not survive a shutdown/restart cycle of the server.

  • The server now handles requests for "favicon.ico" much more gracefully, e.g. it ignores them. Formerly, newer browsers like Firefox would request the icon with each request, and the server always generated the main status page instead. This caused longer delays and wasted CPU time for browsing the HTML interface.

Improved security

  • The server now checks the IP address of the connecting client/proxy against the stored address and mask. Please make sure that your server contains the proper IP info, otherwise client connects will be denied!

  • The client now has the ability to support SSL connections to both the server/proxy and/or the fileserver. Use https://dicop-server:8888/ will automatically encrypt the connection with SSL if both the necessary modules are installed at the client, and the server/proxy supports SSl (config has proto = "SSL" at the server/proxy side, see below).

  • Server, proxy and client now have the ability to support the SSL protocol via the proto = "ssl" setting in the config file. Together with updated clients this allows all communication between server and client to be encrypted.

    However, at the moment the server can only be either ssl or tcp, as indicated by the proto = "foo" setting in the config, e.g. you cannot mix SSL and non-SSL clients (this is a limitation of Net::Server, and currently there is no way to overcome this without rewriting a lot of third-party code from scratch).

    This means that if you switch a server to SSL, all clients must also connecting to that server must also support SSL.

    To overcome this limitation, use your server with proto = "tcp" and then add a proxy to it. Run then a Dicop::Proxy at the same machine (or another machine if desired), and switch that proxy to SSL and point it to your main server as it's upstream server.

    All clients that support SSL must then connect via that proxy, while all others must use the server directly. Here is a picture showing the setup:

            +---------------+   TCP  +---------------+
        | Server (tcp)  |<-------| Proxy (ssl)   |
        +---------------+        +---------------+
                ^                       ^
                | TCP                   | SSL
                |                       |
        +---------------+        +---------------+
        | Client (tcp)  |        | Client (ssl)  |
        +---------------+        +---------------+

    Note that the connection to the file server is independend of the connection to the server/proxy.

    Please see perldoc doc/Dicop.pod for more details.

NEW FEATURES

  • Setting up a complete new server is now made easier with the included script ./setup. Run it to generate the necessary config files, mail and event templates, user and group settings, as well as to change the permissions.

  • Dicop now contains cases, which are just containers to group jobs together. This will help you to manage many jobs more easier. Each job now must belong to exactly one case. If there are no cases upon loading the data, a default case will be created and all jobs added to this case. You can later modify the case, create now ones as well as move jobs from one case to another via the web interface.

    Apart from viewing the case details including a list of jobs belonging to that case, you can also get a complete list of all existing cases via the menu in the footer.

  • New config field case_url_format. Cases (see previous item) have an URL field, and if that URL field is empty, it will be automatically generated from that config key and the current case whenever you view a case via the HTML interface. This means you do not have to type all URLs manually in, and changing them all later on is as easy as editign the config and restarting the server process.

    This feature allows you to link your cases to a case database from DiCoP.

  • New config field send_event_url_format. If defined, the DiCoP server will send events like new job, job finished or result found to that URL by using the templates from tpl/event/.

  • Client's now support the config option sub_arch. If this field is set, the client will append it to the architecture name and send it to the server. For instance, two clients runnning under Linux could have sub_arch set to i386 and ia64 and the server would then receive linux-i386 and >linux-ia64>, respectively.

    The difference would be that the server first tries to serve files from the sub directory named after the sub-architecture, and then one up, and so on. Here is an example for the architecture string 'linux-i386-amd', the server would look into this directories to serve a worker file to the client:

                    worker/linux/i386/amd
                worker/linux/i386
                worker/linux/

    This allows OS or CPU specific overrides to be specified on a per-client basis.

  • A new field in jobtypes allows you to define extra files that need to be downloaded by the client for the worker. The files are architecture-dependend or architecture-independed ('all'). See also above for the new client's sub-arch feature, you can enter as architecture either the base architecture, like linux, or the full architecture string like linux-i386.

    In the first case all clients running under linux would get the extra file, in the second case only these reporting exactly linux-i386 would get it.

  • The list of all open chunks now also shows chunks in the VERIFY state.

  • Entry fields that require a filename (like job or testcase target fields) have a browse button which enables you to select the file from a direcrory view.

  • The client sends now upon failure the output from the worker as error message to the server. The last error message and the time it was generated can be seen on the server's client status page for each client.

  • The server now tracks the time for each connect, and the server status page displays a running average, the overall time and the time it took for the last connect. These times include all the overhead per connect, e.g. not just the time to generate the answer page.

  • Several error messages regarding daemon startup have been made more clear, providing hints on how to fix the problems. A new script called setup will also help for setting up a server for the first time.

  • Support for chunk description files (CDF) is now complete. If extra parameters are necessary on a chunk-by-chunk basis, the server will generate a CDF and tell the client to:

            * download this file
        * pass it to the worker and
        * afterwards delete it

    The client now also supports this fully. This feature is important for special charsets like Dictionary or Extract sets, because these have parameters (like file offsets) that change for every chunk.

    The job description files are still supported and used when their are parameters that are necessary for a job, but do not change for the chunks.

  • In addition to the now fully supported JDF/CDF (see above), the server and client now support "inline" files. For short files (just like CDF/JDF), the server inlines the file data into the answer sent to the client, which extracts and stores it. This saves the client from asking the server for the download location and downloading the file from the file server.

  • It is now possible to disable testcases. Each testcase can be disabled on it's own. This allows you to temp. disable testcases that are known not to work.

KNOWN ISSUES

SSL Support

SSL support does not actually work. We are still investigating why.

Event Posting

Events (job finished, job started etc) will be done while the current request is worked on. If the remote server is down or slow, this may cause the request to be stalled or aborted. Events should be posted outside the main request handling loop, just as emails are being sent.

We plan to implement this soon.

Selecting files/dirs with '_'

It is no longer possible to select files with "_" in the name after you have hit "Reload" on the file selector page.

Dirs without proper permission

These cause an unknown error instead of a more readable "permission denied" when selecting files and directories.

CAVEATS

Obsolete config settings

The config setting is_proxy is obselete and no longer supported. Please remove the appropriate line from your config file. The daemon will warn and refuse to start if it is still present.

Old browser

The HTML interface now uses CSS quite extensively. Old browsers without CSS support (like Netscape 4.x) or with incomplete CSS support (like Konqueror before 3.2, Internet Explorer 5.x etc) will have problems rendering the interface properly. It should be still usable, but it will not be pretty.

Especially in the light of security and looming exploits, we strongly encourage you to upgrade your browser to the newest version.

AUTHOR

(c) Bundesamt fuer Sicherheit in der Informationstechnik 1998-2006

DiCoP is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License version 2 as published by the Free Software Foundation.

See the file LICENSE or http://www.bsi.bund.de/ for more information.