The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
TAP-DOM-Waivers-0.001
River stage zero No dependents
++
/ TAP::DOM::Waivers

NAME

TAP::DOM::Waivers - Patching TAP::DOM, usually for test waivers

SYNOPSIS

 use TAP::DOM;
 use TAP::DOM::Waivers 'waiver';
 
 # get TAP
 my $dom = TAP::DOM->new( tap => "somefile.tap" );
 
 # ,--------------------------------------------------------------------.
 # | Define exceptions and how to modify test results.
 # |
 # | (1) Most powerful but most complex way:
 # |     - use DPath matching and finegrained patching
 # |

 $waivers = [
             {
               # a description of what the waiver is trying to achieve
               comment     => "Force all IPv6 stuff to true",
                  
               # a DPath that matches the records to patch:
               match_dpath => [ "//lines//description[value =~ 'IPv6']/.." ],

               # apply changes to the matched records,
               # here a TODO with an explanation:
               patch       => {
                               is_ok        => 1,
                               has_todo     => 1,
                               is_actual_ok => 0,
                               explanation  => 'waiver for context xyz',
                               directive    => 'TODO',
                              },
             },
            ];
 
 # |
 # | (2) Simpler approach: 
 # |
 # |     - instead of the "patch" key above you can use "metapatches" 
 # |       for Common use-cases, like #TODO or #SKIP
 # |

 $waivers = [
             {
               comment     => "Force all IPv6 stuff to true",
               match_dpath => [ "//lines//description[value =~ 'IPv6']/.." ],
               metapatch   => { TODO => 'waiver for context xyz' },
             },
            ];

 # |
 # | (3) Even simpler:
 # |     - also provide the description as regex
 # |

 $waivers = [
             {
               comment           => "Force all IPv6 stuff to true",
               match_description => [ "IPv6" ],
               metapatch         => { TODO => 'waiver for context xyz' },
             },
            ];
 #
 # |
 # `--------------------------------------------------------------------'

 # the actual DOM patching
 my $patched_tap_dom = waiver($dom, $waivers);
 
 # do something with patched DOM
 use Data::Dumper;
 print Dumper($patched_tap_dom);
 
 # the original DOM can also be patched directly without cloning
 waiver($dom, $waivers, { no_clone => 1 });
 print Dumper(dom);
 
 # convert back to TAP from patched DOM
 print $patched_tap_dom->to_tap;
 print dom->to_tap;

NAME

TAP::DOM::Waivers - Exceptions (waivers) for TAP::DOM-like data

ABOUT

Achieve?

Test waivers are exemptions to actual test results.

This module lets you ignore known issues you don't want to care about, usually by grouping them for a certain context.

Example:

A software project might not run with IPv6 enabled but you want to see a big SUCCESS or NO SUCCESS in an IPv4-only context, without being disturbed by irrelevant IPv6 tests, for now.

Statically marking the problematic tests with #TODO would require to change that back and forth everytime. Dynamically marking those tests depending on the runtime environment does not help when another engineer actually works on fixing those IPV6 problems in the same environment.

The solution is to create a waiver which patches the IPv6 issues away in the results after you actually ran the tests, for later evaluation.

Prove plugin

See also App::Prove::Plugin::Waivers for a way to utilze this module with prove (not yet working?).

Waiver specification

How to match what to patch

This module can patch TAP-DOMs (and similar data structures, see below) by certain criteria. The primary and most powerful way is via Data::DPath paths, as it allows to match fuzzily against continuously changing TAP from evolving test suites.

I use this with a big TAP database where I activate waivers as a layer on top of TAP::DOM based evaluation. There the TAP-DOMs are just part of a even bigger data structure, but the DPath matching still applies there.

match_dpath => [ @array_of_dpaths ]

This provides a set of dpaths that are each tried to match. The DPaths should point to a single entry in TAP-DOM - that's why the examples above go down into an entry to match conditions (like the description), and then go up one level to point to the whole entry.

match_description => [ @array_of_regexes ]

This is a high level frontend to match_dpath. The regexes are internally embedded in dpaths which are then used to match. The converted internal dpaths will match fuzzy for a typical TAP-DOM structure, in particular:

 "//lines//description[value =~ qr/$description/]/..";

Please note that this doesn't allow to specify complex conditions like the combination of a description and a particular test success (e.g. only the "not ok" tests with a particular description, see examples in t/waivers.t).

In combination with the also just canonically working metapatch (see below) it might create a slightly different TAP-DOM than you expect, e.g. when you match and modify tests as '#TODO' that did not even fail, but the metapatch marks them as 'not ok #TODO'. So the original actual success is lost.

It might be still "quite ok" and worth the less complexity but consider using match_dpath for better control.

Patch specs

patch => { %patch_spec }

A hash entry key patch contains single keys that overwrite respective fields of a TAP-DOM entry.

This allows finegrained control but it's somewhat difficult if you are not familiar with the details of how a TAP situation looks like in a TAP-DOM.

Therefore you can describe more abstract use-cases with metapatches.

metapatch => { %patch_spec }

A key metapatch declares a common use-case. Inside a metapatch the key describes the use case (like 'TODO'), and the value is the most significant thingie (eg. the explanation).

Currently these metapatches are supported:

  • TODO => explanation

  • SKIP => explanation

When such a metapatch is found it is converted internally into an equivalent detailed patch, as described above.

Comments

The key comment is not strictly needed. It will help once there is some logging.

Back from DOM to TAP

Usually you regenerate a semantically comparable TAP document from the DOM via TAP::DOM::to_tap.

API

waive ($dom, $waivers, $options)

This applies a set of waivers to a TAP-DOM.

The TAP-DOM is usually a real TAP::DOM but don't have to. It is explicitely allowed to provide similar data structures, e.g., bigger structures that only contain TAP-DOMs in sub structures. It's your responsibility to provide something meaningful.

If you match with match_dpath you have control whether to use the surrounding data structures to match or not.

If a waiver does not match, nothing happens.

AUTHOR

Steffen Schwigon, <ss5 at renormalist.net>

BUGS

Please report any bugs or feature requests to bug-tap-dom-waivers at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=TAP-DOM-Waivers. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

SUPPORT

You can find documentation for this module with the perldoc command.

    perldoc TAP::DOM::Waivers

You can also look for information at:

LICENSE AND COPYRIGHT

Copyright 2011 Steffen Schwigon.

This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

See http://dev.perl.org/licenses/ for more information.

AUTHOR

Steffen Schwigon <ss5@renormalist.net>

COPYRIGHT AND LICENSE

This software is copyright (c) 2014 by Steffen Schwigon.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.