Test::Stream::Design - Design documentation
This document covers some of the important design decisions that were made in Test::Stream. Each sections covers a concept that needed to be addressed, or a problem that needed to be solved. Along with the description of the issue, and requirements, is a high level overview of the design adopted. In some cases alternative possible implementations are listed.
Each section follows this format as closely as it can.
=head1 Short Title Problem Statement =head2 Specification =head2 Implementation Proposals =head3 Selected proposal details Pros: ... Cons: ... =head 3 Alternate Proposal details Pros: ... Cons: ...
Please also try to limit implementation proposal code to be just enough to convey the idea. The code should be flexible and adaptable to backcompat needs and reality.
Legacy Test::Builder had a bad habbit of breaking encapsulation, and even encouraging external tools to do the same. Moving forward we want to use proper encapsulation. That said, we need to keep an eye on performance. Object methods, specially when implemented poorly, are significantly slower than direct element access. To worry too much about that in advance would be premature, however the Test::Stream prototype already showed that we do need to concern ourselves with it as performance quickly becomes an issue when we do not.
We cannot use Moose, or any other non-core object system. This was dictated as far back as the Test::Builder2 prototypes. We need something simple, and bundled. We do not want to duplicate constructor code in every object. We want all objects to be consistent in both constructors and accessors.
The object system should give us a constructor, and allow us to specify attributes by name. The object system must be dead simple, we have no intention of duplicating Moose or its extensive functionality. Roles and anything more than read/write accessors would be overkill.
Note This is the one used by Test::Stream
use HashBase( accessors => [qw/foo bar baz/], base => 'My::Base::Class', ); my $self = __PACKAGE__->new(foo => 1); # Constructor my $val = $self->foo; # Reader $self->set_foo($val); # Writer $val = $self->{+FOO}; # Safe internal direct hash read $self->{+FOO}; # Safe internal direct hash write
Seperate reader and writer methods are faster that a combined read/write accessor. Seperate methods also make intentions more clear.
Generates accessors are very minimal:
sub foo { $_[0]->{ATTRIBUTE} }; sub set_foo { $_[0]->{ATTRIBUTE} = $_[1] };
Constructor takes a list of key/value pairs and blesses a hashref with the pairs. There may be some extra safety checks to ensure all keys passed in are valid attributes.
$val = $self->{+FOO}; $self->{+FOO};
This is perfectly reasonable in internal object code. The constants avoid typo mistakes, and we avoid the overhead of method calls. External code should use methods to avoid breaking encapsulation.
The meta-object prevents you from adding duplicate attributes, allows subclasses to inherit attribute constants, and makes the constructor faster when it validates the attributes you specify.
The Test::Stream implementation can be seen in Test::Stream::HashBase and Test::Stream::HashBase::Meta.
(This needs a better title)
When an assertion such as ok(1) is written in a test file, there is state and context information that is important for diagnostics.
ok(1)
Standalone tools usually do not have a problem, they can get the immediate caller, and any additionl info without any hurdles. This is normally convenient, but problematic when you attempt to wrap the tool. Legacy Test::Builder used the $Level variable for situations like this. Unfortunately most people did not know about the level variable, or how to use it properly. There are also some situations where $Level is not sufficient.
$Level
This is a simple example to demonstrate the problem:
10: sub my_ok { 11: my ($x, $y) = @_; 12: ok($x, 'x is good'); 13: ok($y, 'y is good'); 14: } 15: 16: my_ok(1, 0); 17: my_ok(0, 1);
In this example, the failure on line 16 would report to line 13. The failure on line 17 would report to line 12. We need a simple way to have the outermost tool find and lock-in the diagnostics information for all inner-tools to find and use. In this case we want my_ok() to get the caller and lock it in, ok() would then find this locked-in information instead of using caller.
my_ok()
ok()
my_ok() would aneed to clear the information when it is done, inner tools would have to avoid clearing the information. This is where the use of local on $Level was useful in legacy code.
local
State that may be useful to capture and convey:
The context object is a structure that holds a snapshot of the important data. This structure can be obtained at any point by any tool that needs it. If an existing snapshot is present it should be returned. If no snapshot is present one should be generated. Consumers of the snapshot must not be required to take a stack trace of their own as they do with $Level.
For simplicity context should be obtained using a function called context() that does not require any parameters (Though it can have optional parameters). When a context already exists it should be returned. When none exists a new one will be generated and returned. To establish the context, the entry sub for any tool should obtain the context first thing.
context()
Note: This is the implementation Test::Stream currently uses.
Psuedo-perl:
sub snapshot { my $self = shift; return ...shallow copy of context... } my $CONTEXT; sub context { return $CONTEXT if $CONTEXT; my $ctx = ...generate... $CONTEXT = $ctx; weaken($CONTEXT); return $ctx; }
Consumer:
sub my_ok { my $ctx = context(); my ($x, $y) = @_; ok($x, 'x is good'); ok($y, 'y is good'); }
In this implementation, the first tool to ask for a context generates it. Any later tools find the existing context. When each tool completes, the context ref is uses goes away. When the root tool completes the final context reference will be collected and $CONTEXT is automatically set to undef.
$CONTEXT
Pros:
Cons:
This can be mitigated with a snapshot() method that returns a shallow copy of the context. However it is very important that nobody store the original context perminently.
snapshot()
The current implementation can be seen in Test::Stream::Context.
sub snapshot { my $self = shift; return ...shallow copy of context... } my $CONTEXT; sub release { my $self = shift; $CONTEXT = undef if $CONTEXT == $self; } sub context { return $CONTEXT->snapshot if $CONTEXT; my $ctx = ...generate... $CONTEXT = $ctx; return $ctx; }
sub my_ok { my $ctx = context(); my ($x, $y) = @_; ok($x, 'x is good'); ok($y, 'y is good'); $ctx->release; }
In this implementation, the first tool to request a context generates it, and gets the original. Any future consumers get a copy of the snapshot. Each tool calls 'release' on the context it has when it is done with it. For snapshots the release method is a no-op. For the original context the release method will actually release it.
my $CONTEXT; sub context(&) { my $code = shift; my $unset = 0; unless($CONTEXT) { $unset = 1; $CONTEXT = ...Generate...; } local $@; my $lives = eval { $code->($CONTEXT); 1; }; my $error = $@; $CONTEXT = undef if $unset; die $error unless $lives; }
sub my_ok { my ($x, $y) = @_; context { my $ctx = shift; # if you want it ok($x, 'x is good'); ok($y, 'y is good'); }; }
This was the first solution used in the initial refactors. This system required all test tools to be marked as test tools through some form of metadata. Any time some context info such as $TODO or the place to report errors was needed a full stack trace would be used. This slowed down test suites by 300%. There was an XS implementation that made it reasonably fast, but it was clearly the wrong way to go.
$TODO
Many testing tools add behavior when events (such as ok, diag, note, plan, etc) occur. In the legacy code the only ways to accomplish this involve breaking encapsulation, either by monkeypatching or replacing the Test::Builder singleton. There is enough demand for hooking into these events that we need a formal API for doing so, one that preserves encapsulation.
Several events will be defined. Each event will be a fully encapsulated object. In order to make an assertion or report diagnostics information, you must create an event object. All event objects will be handed off to a processor which handles the rest.
The processor has 2 jobs, synchrnization, and hooks. The process manages state, and directs events to where they need to go. In this system the processor is the only component that acts as a singleton. The processor should be as minimal as possible to avoid a monolithic singleton.
There should be no restrictions against creating multiple instances of the processor object. There should also be a way to get a canonical processor to which events should be sent by default. In other words this is where a singleton or similar is likely to be necessary.
There should be 3 primary hooks into the stream of events passing through the processor:
This hook allows tools to modify event objects as soon as they enter the hub, before they effect test state or become output.
This hook allows tools to monitor events. Tools use this hook to see a copy of all events after they have been munged, effected state, and been output to TAP (if tap is enabled).
This hook is called at the end of testing. This can be seen as a hook into done_testing(), but it will also trigger at the end of the tests if done_testing is not used. This is called before done_testing determined how many tests have run, so it is safe to send additional events in this hook.
done_testing()
done_testing
Test::Builder showed us that having a monolithic singleton is a bad idea. It also showed us that making the variable that holds the singleton globally accessible is a mistake. We do need something like a singleton though, all producers need to send their events somewhere.
The hub stack model looks like this:
{ my @HUB_STACK; # lexical and scoped so that it cannot be directly modified from the outside # This is how producers get the canonical hub they should be using sub shared { push @HUB_STACK => Hub->new unless @HUB_STACK; return $HUB_STACK[-1] } sub intercept { my $hub = shift; push @HUB_STACK => $hub; } sub unintercept { my $hub = shift; die unless $hub == $HUB_STACK[-1]; pop @HUB_STACK; } }
Producers:
my $event = ...; shared()->send($event);
There are valid reasons to replace the hub, this is why it is a stack instead of a single scalar. Whatever hub is at the top of the stack will recieve all events. The primary reason to push a hub on the stack is test validation. Using this system you can replace the hub temporarily to intercept all events and validate that they are what you expect.
The hub itself is pretty minimal. Get objects via $hub->send($e), each hook has a chance to do something with the event, and TAP is optionally output. Hooks will accept coderefs, and any number of coderefs can be given to a hook.
$hub->send($e)
$hub->munge(sub { my ($e) = @_; ... });
For most tools the only thing they need is the ability to fire off events. Test::Builder actually got this right with a single object and methods for each event. This API is hard to improve upon, so we need to focus on not making it harder.
A single easy to obtain object with methods to quickly fire off events. Assuming the use of the context model, the context object can be what we need. The context object is necessary for all events anyway, so it can have "shortcut" methods that mirror the event methods Test::Builder had.
Note Test::Stream uses this design.
This proposal assumes the use of the context model mentioned above.
my $tx = context(); $ctx->ok(1, "an example"); $ctx->diag("This is an example");
In the context model, all events will need a copy of, or reference to the context. Using the context object that you already have anyway as the API for generating events makes sense.
Some tools run several test files from a single process. Examples include the well known Test:::Class. Sometimes you want different setting in each file, such as having a different output encoding. Other examples include test files that validate testing tools and need to replace fielhandles for just their own scope.
Simple way for Test::Stream, and other test tools, to store and retrieve file(package) specific metadata.
A meta-object automatically created for all packages that load a testing tool. It will vivify as needed. Tools that need to track meta-data can all use the same one instead of writing their own independant meta-objects.
Test::Stream implements this in Test::Stream::Meta.
Test::Builder clones STDERR and STDOUT as soon as it loads. Once the handles are cloned it transforms them and then stores them. All output from Test::Builder is sent to these cloned handles. A result of this is that setting an encoding on STDERR and STDOUT has no effect. Setting the encoding involves jumping through hoops to modify the handles before loading Test::Builder, or altering Test::Builders copies of the filehandles.
We need to preserve backwards compatability, and doing so means preserving the default behavior listed above. However we need an easy way to switch encodings as needed.
In this model there is an object that manages all the filehandles and encodings. For each encoding it has the 3 filehandles Test::Builder expect, OUT, ERR, and TODO. Initially there is one set of filehandles called 'legacy', these do what Test::Builder currently does. Other encodings such as 'utf8' can also be initialized, and will hold the 3 expected filehandle clones.
Each test package will have meta-data associated with it that includes the encoding to use when tap is generated from that package. By default every package uses the 'legacy' filehandles. There should be both a utility function, and an import argument, that let someone specify an alternate encoding to use in the test package.
The current Test::Stream implementation of this can be seen in Test::Stream itself, the IO manager is Test::Stream::IOSets.
The Test::Stream meta object can be found at Test::Stream::Meta.
Subtests require independant state tracking. All testing tools need to work as expected within subtests. In Test::Builder this was a huge pain that required a confusing and fragile hack. This should not be hard, and it should not be a hack.
The 'hub' object which receives all events and synchronizes state should track state in an encapsulated object. To take it further the hub object will have a stack of state objects, and any event seen will modify the state on top of the stack. Subtests will push a new state object when they start, and pop the state object when they finish.
The state object can be found in Test::Stream::State.
Test::Builder supports threading using shared variables. There are many conditions that can break shared variables, such as overloaded objects. In addition the shared variables do not work for tests that need to fork. There are a lot of cpan modules that implement forking support, but none are highly promoted.
Forking, or starting new threads, then writing assertions in each one should just work. At the very least it should not require more than a simple 'on' switch. People writing tests should not need to worry about synchronizing things on their own.
The main complications with concurrency are:
Test::Stream requires you to request concurrency. You can turn concurrency on either by loading threads before Test::Stream, or by an import argument. This is done because the concurrency support typically doubles startup time.
When concurrency is turned on, a temporary directory is created for IPC. All events in the parent process are handled as normal. All events recieved in a child process are forwarded to the parent. To forward events, the child writes them to files in the temporary directory. The parent will frequently cull results from the temp directory and process them as it would any other event.
Subtests are special. When you start a subtest in a child process, all events are handled by the subtest in the child. Once the subtest is complete the final subtest event itself is sent to the parent. This works fine as subtest output is not actually parsed in any way, only the final subtest event itself matters.
The final consideration is what to do when forking occurs inside a subtest. In Test::Stream the subtest itself is forked, so both processes end up sending a final subtest event.
All the logic for this is encapsulated in the Test::Stream:Hub object.
Validating test tools is difficult at best. Validation requires that TAP output and state changes are suspended and intercepted. After interception you need a way to check that they are what you expect. You then need to output TAP and modify state for real.
Test::Builder::Tester intercepts the TAP and makes you compare strings. This is fragile as any change to output, including comments, can break any number of downstream tools. Test::Tester is also limited in that it does not support validating all events.
It should be trivial to suspend and intercept events. There should be tools that make it easy to confirm the events are what you expect. You should not be limited to string comparisons. In fact validation of TAP output should not be the test-tools job in the first place.
my $events = intercept { ... }; events_are($events, ..., "Events are what we expect");
See Test::Stream::Tester for the implementation, it provides a lot more than is seen in this example. Under the hood this uses the hub-stack model to temporarily swap-in a second Test::Stream::Hub implementation. Using this it is even possible to validate the test-testing tool using itself.
Test::Builder has a lot of magic that it runs at the end of testing. Most of this magic occurs in an END block. There is a compelling argument that nearly all of this exit magic actually belong in the test harness. Backwords compatability requirements dictate that we need to preserve this behavior, regardless of where it actually belongs.
Existing behaviors to preserve:
Specifically Test::Builder sets the exit status to the number of failed tests, or 255, whichever is smaller.
If no plan was output, or done_testing was not called, it will warn you.
Test::Stream moved all the END logic into the Test::Stream::ExitMagic module. An instance of this module is created by Test::Stream when the root hub is initialized. When tests are done the object is told to do its thing in an END block. The ExitMagic module always operates on the root Test::Stream::ExitMagic object, not the one on the top of the stack.
The following people have all contributed to this document in some way, even if only for review.
The source code repository for Test::More can be found at http://github.com/Test-More/test-more/.
Copyright 2015 Chad Granum <exodist7@gmail.com>.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
See http://www.perl.com/perl/misc/Artistic.html
To install Test::Simple, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Test::Simple
CPAN shell
perl -MCPAN -e shell install Test::Simple
For more information on module installation, please visit the detailed CPAN module installation guide.