NAME

Test::Rail::Parser - Upload your TAP results to TestRail

VERSION

version 0.049

DESCRIPTION

A TAP parser which will upload your test results to a TestRail install. Has several options as to how you might want to upload said results.

Subclass of TAP::Parser, see that for usage past the constructor.

You should probably use App::Prove::Plugin::TestRail or the bundled program testrail-report for day-to-day usage... unless you need to subclass this. In that case a couple of options have been exposed for your convenience.

CONSTRUCTOR

new(OPTIONS)

Get the TAP Parser ready to talk to TestRail, and register a bunch of callbacks to upload test results.

OPTIONS - HASHREF -- Keys are as follows:

apiurl - STRING: Full URI to your TestRail installation.

user - STRING: Name of your TestRail user.

pass - STRING: Said user's password, or one of their valid API keys (TestRail 4.2 and above).

debug - BOOLEAN: Print a bunch of extra messages

browser - OBJECT: Something like an LWP::UserAgent. Useful for mocking.

run - STRING: name of desired run.

plan - STRING (semi-optional): Name of test plan to use, if your run provided is a child of said plan.

configs - ARRAYREF (optional): Configurations to filter runs in plan by. Runs can have the same name, yet with differing configurations in a plan; this handles that odd case.

project - STRING (optional): name of project containing your desired run. Required if project_id not passed.

project_id - INTEGER (optional): ID of project containing your desired run. Required if project not passed.

step_results - STRING (optional): 'internal name' of the 'step_results' type field available for your project.

result_options - HASHREF (optional): Extra options to set with your result. See TestRail::API's createTestResults function for more information.

custom_options - HASHREF (optional): Custom options to set with your result. See TestRail::API's createTestResults function for more information. step_results will be set here, if the option is passed.

testsuite - STRING (optional): Attempt to create a run based on the testsuite identified by the name passed here. If plan/configs are passed, create it as a child of said plan with the listed configs. If the run exists, use it and disregard this option. If the containing plan does not exist, create it too. Mutually exclusive with 'testsuite_id'.

testsuite_id - INTEGER (optional): Attempt to create a run based on the testsuite identified by the ID passed here. If plan/configs are passed, create it as a child of said plan with the listed configs. If the run exists, use it and disregard this option. If the plan does not exist, create it too. Mutually exclusive with 'testsuite'.

sections - ARRAYREF (optional): Restrict a spawned run to cases in these particular sections.

autoclose - BOOLEAN (optional): If no cases in the run/plan are marked 'Untested' or 'Retest', go ahead and close the run. Default false.

encoding - STRING (optional): Character encoding of TAP to be parsed and the various inputs parameters for the parser. Defaults to UTF-8, see Encode::Supported for a list of supported encodings.

test_bad_status - STRING (optional): 'internal' name of whatever status you want to mark compile failures & no plan + no assertion tests.

max_tries - INTEGER (optional): number of times to try failing requests. Defaults to 1 (don't re-try).

In both this mode and step_results, the file name of the test is expected to correspond to the test name in TestRail.

This module also attempts to calculate the elapsed time to run each test if it is run by a prove plugin rather than on raw TAP.

The constructor will terminate if the statuses 'pass', 'fail', 'retest', 'skip', 'todo_pass', and 'todo_fail' are not registered as result internal names in your TestRail install.

The purpose of the retest status is somewhat special, as there is no way to set a test back to 'untested' in TestRail, and we use this to allow automation to pick back up if something needs re-work for whatever reason.

The global status of the case will be set according to the following rules:

    1. If there are no issues whatsoever besides TODO failing tests & skips, mark as PASS
    2. If there are any non-skipped or TODOed fails OR a bad plan (extra/missing tests), mark as FAIL
    3. If there are only SKIPs (e.g. plan => skip_all), mark as SKIP
    4. If the only issues with the test are TODO tests that pass, mark as TODO PASS (to denote these TODOs for removal).
    5. If no tests are run at all, and no plan made (such as a compile failure), the cases will be marked as failures unless you provide a test_bad status name in your testrailrc.

Step results will always be whatever status is relevant to the particular step.

TAP Extensions

Forcing status reported

A line that begins like so:

% mark_status=

Will allow you to force the status of a test case to whatever is on the right hand side of the = expression.

Example (force test to retest in event of tool failure):

    my $failed = do_something_possibly_causing_tool_failure();
    print "% mark_status=retest" if $failed;

Bogus statuses will cluck, but otherwise be ignored. Valid statuses are any of the required internal names in your TestRail install (see above).

Multiple instances of this will ignore all but the latest valid status.

PARSER CALLBACKS

unknownCallback

Called whenever we encounter an unknown line in TAP. Only useful for prove output, as we might pick a filename out of there. Stores said filename for future use if encountered.

commentCallback

Grabs comments preceding a test so that we can include that as the test's notes. Especially useful when merge=1 is passed to the constructor.

testCallback

If we are using step_results, append it to the step results array for use at EOF. Otherwise, do nothing.

bailoutCallback

If bail_out is called, note it and add step results.

EOFCallback

If we are running in step_results mode, send over all the step results to TestRail. Otherwise, upload the overall results of the test to TestRail.

planCallback

Used to record test planning messages.

make_result

make_result has been overridden to make the parser object available to callbacks.

NOTES

When using SKIP: {} (or TODO skip) blocks, you may want to consider naming your skip reasons the same as your test names when running in test_per_ok mode.

SEE ALSO

TestRail::API

TAP::Parser

SPECIAL THANKS

Thanks to cPanel Inc, for graciously funding the creation of this module.

AUTHOR

George S. Baugh <teodesian@cpan.org>

SOURCE

The development version is on github at https://github.com/teodesian/TestRail-Perl and may be cloned from git://github.com/teodesian/TestRail-Perl.git

COPYRIGHT AND LICENSE

This software is copyright (c) 2021 by George S. Baugh.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

cpanm TestRail::API

perl -MCPAN -e shell
install TestRail::API