The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
Linux-Slackware-SystemTests-0.01
River stage zero No dependents
++
/ Linux::Slackware::SystemTests

NAME

Linux::Slackware::SystemTests - System tests for Slackware Linux

SYNOPSIS

  # If you just want to -run- system tests, invoke the slackware-systemtests
  # test harness and stop reading this document:

  $ slackware-systemtests

  # If you are -writing- system tests, use an instance to get useful tools:

  use Linux::Slackware::SystemTests;
  my $st = Linux::Slackware::SystemTests->new();

  # Copy a data file from wherever they are installed to a temp directory so
  # it can be modified:

  my ($ok, $file_pathname) = $st->init_work_file("001_sed.1.txt");

  # $st->{sys_hr} provides useful details about the system being tested, so
  # you can change the test depending on what version of Slackware is being
  # tested, or 32-vs-64-bit, etc:

  if ($st->{sys_hr}->{version} eq "14.2") {
      # run test for just the Slackware 14.2 release
  }
  if ($st->{sys_hr}->{bits} eq "64") {
      # run test specific to 64-bit Slackware
  }

  # If you need to JSONify something, there's already a JSON object
  # instantiated with sane and robust operating parameters:

  ok `/bin/uname -a` =~ /CPU \@ ([\d\.]+)GHz (\w+)/, "processor is ".$st->{js_or}->encode([$1, $2]);

  # If you need to know where the test data files are installed, $st knows:

  my $data_pathname = "$st->{data_dir}/some_test_file.txt";

  # If you want to log structured data to a file, that can be done too:
  $st->log("WARNING", "Something not quite right with environment", \%ENV);

  # Alternatively, if your test needs none of these things, you don't have to
  # use this module at all!  Any test that produces TAP output should jfw.

DESCRIPTION

System tests are are short programs which exercise components of your computer system and make sure they are running correctly.

This package implements tests for Slackware Linux systems, and a test harness for running them and generating reports.

This module provides maybe-useful functionality for making those tests easier to write, and for helping to write test harnesses.

The eventual goal is to accumulate enough tests that when Slackware updates, you can just re-run the system tests and know that everything works okay. Some sort of continuous integration automation might also happen, eventually.

Another goal is to make it easy to write new system tests, to keep the barrier of entry low. There is a lot to test in a full Slackware install, so please write tests and share them with the author :-)

USING THE MODULE

The Linux::Slackware::SystemTests module provides some data and methods which may or may not be useful to you.

If you do not find them useful, don't use the module! :-) System tests do not require the module. It's just there to make your life easier, if you need it.

METHODS

my $st = Linux::Slackware::SystemTests->new(%options)

Instantiates a new Linux::Slackware::SystemTests object. Lacking options, sane defaults are assumed.

Supported options are:

temp_dir => (path string)

Set this to override $st->{temp_dir}, which changes the behavior of the init_work_file and invoke_self_id methods.

When this option is not provided, an appropriate temporary directory will be found via File::Valet::find_temp.

Some sane values to pass here might be "/tmp" or "/dev/shm".

debug => (0 or 1)

When true (1), invoking $st->log with a mode of "DEBUG" will be silently ignored.

When false (0), "DEBUG" mode logging messages will be written to the log file.

Defaults to false (0).

log_filename => (pathname string)

Sets the pathname of the structured data log file written to by $st->log. Defaults to "/tmp/st.log".

show_log => (0 or 1)

When true (1), invoking $st->log will cause a human-friendly representation of the log record to be written to STDOUT.

When false (0), no log records will be displayed.

Defaults to false (0).

log => (0 or 1)

When true (1), invoking $st->log will cause log messages to be written to the logile.

When false (0), no log records will be written to file, but might still be written to STDOUT if show_log is set.

Defaults to true (1).

share_dir => (path string)

Set this to override $st->{share_dir}, the base directory from which data_dir, bin_dir, tests_dir are derived. Mostly useful for mocking purposes.

data_dir => (path string)

Set this to override $st->{data_dir}, where the test data files are stored. Mostly useful for mocking purposes.

tests_dir => (path string)

Set this to override $st->{tests_dir}, where the system test executables are stored. Mostly useful for mocking purposes.

bin_dir => (path string)

Set this to override $st->{bin_dir}, where the module's private executables are stored. Mostly useful for mocking purposes.

self_id => (hash reference)

Set this to override $st->{sys_hr}. Mostly useful for mocking purposes.

When this option is not provided, a self-id script will be run which figures out various details about the local system.

$st->log(MODE, DESCRIPTION[, structured data ...])

Writes a JSON record to a structured data log file, and optionally to STDOUT as well.

MODE

Should be one of "DEBUG", "ERROR", "WARNING", "FAIL", or "PASS".

DESCRIPTION

Should be an invariant string (without any variables interpolated into it). Using an invariant makes possible a full enumeration of log record types, which is important for log collation.

structured data

Can be any number of arbitrarily complex data elements. Elements not able to be represented by the JSON module will instead be represented as nil. This includes code refs, glob refs and regex refs.

Each JSON record is newline-terminated, and contains the following fields:

  [MODE, EPOCH_TIME, LOCAL_TIME, PID, DESCRIPTION, structured data ...]

The MODE, DESCRIPTION and structured data fields will be the JSON representations of the $st->log parameters.

The EPOCH_TIME field will be a floating point representation of the epoch time at which the log record was created.

The LOCAL_TIME field will be a human-readable representation of EPOCH_TIME in the local timezone.

The PID field will be the process identifier of the process which created the log record.

For instance, the following log call:

    $st->log("WARNING", "skipped some tests", {name => "ichi", why => "ploo"}, {name => "ni", why => "glom"}, [1, 2, 3, {foo => "bar"}])

.. would append something like the following JSON to the logfile:

    ["WARNING", 1470253241.25485, "Wed Aug  3 12:40:41 2016", 1472, "skipped some tests", {"name": "ichi", "why": "ploo"}, {"name": "ichi", "why": "glom"}, [1, 2, 3, {"foo": "bar"}]]

Furthermore, if the show_log parameter was set when $st was instantiated, the following would be printed to STDOUT:

    1470253241.25485 Wed Aug  3 12:40:41 2016 1472\tWARNING\t["skipped some tests", {"name": "ichi", "why": "ploo"}, {"name": "ichi", "why": "glom"}, [1, 2, 3, {"foo": "bar"}]]

($ok, $pathname) = $st->init_work_file("314_some_test_file.txt")

When the module is installed, it is usually stored as read-only data in some obscure corner of the filesystem. This is inconvenient when a test requires a data file which is writable.

Rather than forcing each test author to come up with a way to find the data and copy it to a temporary directory (which might not exist on the system), init_work_file is provided to do the work for them.

init_work_file will find the data file, find a temporary directory, delete any old files left over from a previous run, copy the file and return ('OK', $pathname) where $pathname is the full pathname of the copied file.

If it encounters any errors at any point in the process, it will return ('ERROR', $description) where $description describes what failed and (maybe) why.

If a copy of the file is not required, and a test only needs the full pathname of a data file for reading, use $st->{data_dir} instead, like so:

    my $full_pathname = "$st->{data_dir}/314_some_test_file.txt";

$st->opt(OPTION_NAME[, DEFAULT_VALUE[, ALTERNATIVE_HASHREF]])

    $st->opt("log")
    $st->opt("log", 0)
    $st->opt("log", 0, $alt_hr)

Fetches an option field from the object's instantiation parameters.

opt will look in $st->{opt_hr}->{OPTION_NAME} first. If not present there, it will look in $alt_hr->{OPTION_NAME} if an $alt_hr parameter was provided.

If no option by that name is found anywhere, opt will return DEFAULT_VALUE (0 in the above examples), or undef if no default is provided.

WRITING SYSTEM TESTS

System tests can be very simple or as complicated as necessary. They may be written in any language, although /bin/sh and perl are encouraged.

The only hard requirement is that they generate their output in Test Anything Protocol, which is pretty easy. TAP libraries are available for most languages.

Tests should be executable files located in lib/Linux/Slackware/SystemTests/system_tests/ with a .t filename suffix. All such files will be executed by running the slackware-system-test script.

WRITING SYSTEM TESTS IN PERL

Writing tests in perl is easy. Just copy lib/Linux/Slackware/SystemTests/system_tests/test_template.pl to a new file (like 123_my_test.t) and edit the new file to add your test logic. There are some goodies in test_template.pl (like object instantiation) which are commented out. Uncomment them if you need them.

test_template.pl uses Test::Most, but feel free to use any of the other TAP-compliant test modules, such as Test::More or Test2.

If you have never written tests for perl before, read the Test::Most documentation and look at the other .t files in the system_tests directory to get a notion.

The skinny of it is, Test::Most provides functions like ok and is, to which you pass the results of your tests of system correctness, and it represents those results in TAP format. For instance:

    ok `lsmod 2>\&1` =~ /ipv6/, "IPv6 module is loaded"

.. which displays ok 1 - IPv6 module is loaded or not ok 1 - IPv6 module is loaded depending on the results of the expression.

Also feel free to hop onto the #perl IRC channel on irc.freenode.org to ask for help. The good folks there are very enthusiastic about good tests. Just don't take mst's brisk manner personally. He means well.

WRITING SYSTEM TESTS IN BASH

Work in progress. More here later.

I'm still figuring this out. There is a TAP implementation for bash test-more-bash which might be appropriate, but I'm still assessing it.

If that doesn't work out, I'll teach slackware-system-test to accept *.sh.t tests which signal pass/fail with an exit code, and drop the TAP requirement. The priority is to get more tests written, and barriers will be lowered to make that happen.

WRITING SYSTEM TESTS IN OTHER LANGUAGES

Work in progress. More here later.

Test Anything Protocol claims TAP libraries are available for C, C++, Python, PHP, Perl, Java, JavaScript, "and others", which means whatever programming language you like to use, you can likely use it to write system tests.

The only stipulation is that the code should jfw using software that ships with stock Slackware. Since gcc and g++ are part of Slackware, C and C++ are fine, but Oracle's JVM does not. That means unless your test works with gcc-java, Java is off the table.

RUNNING SYSTEM TESTS

At the moment, the test harness is extremely simple. More features will come. The main priority is getting more tests written.

For the moment, invoking slackware-system-test without parameters will cause it to run all of the *.t executables in system_tests, display their pathnames, and display only those tests which fail.

Invoking slackware-system-test with arguments will treat those arguments as regex patterns which will be applied to the names of the *.t executables in system_tests, and only those which match will be executed.

Thus if system_tests contains tests 123_ichi.t, 234_ni.t and 345_san.t, running slackware-system-test s will cause only 345_san.t to run, while running slackware-system-test i will cause only 123_ichi.t and 234_ni.t to run.

Alternatively, to run specific system tests, invoke them directly:

    $ lib/Linux/Slackware/SystemTests/system_tests/001_sed.t

Near future plans include a --retry option which only runs tests which failed in the previous invocation and some sort of html report output.

Far future plans include continuous integration automation, so that new releases of Slackware can be installed to a VM and tested, and test results made available as a web page.

SEE ALSO

The Linux Testing Project which does not work under Slackware and has more of a kernel focus.

CONTACTS AND RESOURCES

Github page https://github.com/ttkciar/linux-slackware-systemtests is the official project site. Submit bug reports and pull requests there (or just email TTK).

Channel ##slackware on irc.freenode.net, for Slackware-related questions

Channel #perl on irc.freenode.net, for Perl-related questions

AUTHORS

Contribute some system tests and get yourself added to this list!

TTK Ciar, ttk@ciar.org

COPYRIGHT AND LICENSE

Copyright (C) 2016, TTK Ciar and others.

This program is free software, you can redistribute it and/or modify it under the terms of Perl itself.