Test::Against::Dev - Test CPAN modules against Perl dev releases
my $self = Test::Against::Dev->new( { application_dir => '/path/to/application', } ); my ($tarball_path, $work_dir) = $self->perform_tarball_download( { host => 'ftp.funet.fi', hostdir => /pub/languages/perl/CPAN/src/5.0, perl_version => 'perl-5.27.6', compression => 'gz', work_dir => "~/tmp/Downloads", verbose => 1, mock => 0, } ); my $this_perl = $self->configure_build_install_perl({ verbose => 1, }); my $this_cpanm = $self->fetch_cpanm( { verbose => 1 } ); my $gzipped_build_log = $self->run_cpanm( { module_file => '/path/to/cpan-river-file.txt', title => 'cpan-river-1000', verbose => 1, } );
This library should be used by anyone who wishes to assess the impact of month-to-month changes in the Perl 5 core distribution on the installability of libraries found on the Comprehensive Perl Archive Network (CPAN).
This problem is typically referred to as Blead Breaks CPAN (or BBC for short). Perl 5 undergoes an annual development cycle characterized by monthly releases whose version numbers follow the convention of 5.27.0, 5.27.1, etc., where the middle digits are always odd numbers. (Annual production releases and subsequent maintenance releases have even-numbered middle digits, e.g., 5.26.0, 5.26.1, etc.) A monthly development release is essentially a roll-up of a month's worth of commits to the master branch known as blead (pronounced "bleed"). Changes in the Perl 5 code base have the potential to adversely impact the installability of existing CPAN libraries. Hence, various individuals have, over the years, developed ways of testing those libraries against blead and reporting problems to those people actively involved in the ongoing development of the Perl 5 core distribution -- people typically referred to as the Perl 5 Porters.
5.27.0
5.27.1
5.26.0
5.26.1
This library is intended as a contribution to those efforts. It is intended to provide a monthly snapshot of the impact of Perl 5 core development on important CPAN libraries.
Unlike other efforts, Test-Against-Dev does not depend on test reports sent to CPANtesters.org. Hence, it should be unaffected by any technical problems which that site may face. As a consequence, however, a user of this library must be willing to maintain more of her own local infrastructure than a typical CPANtester would maintain.
While this library could, in principle, be used to test the entirety of CPAN, it is probably better suited for testing selected subsets of CPAN libraries which the user deems important to her individual or organizational needs.
This library is currently focused on monthly development releases of Perl 5. It does not directly provide a basis for identifying individual commits to blead which adversely impacted particular CPAN libraries. It "tests against dev" more than it "tests against blead" -- hence, the name of the library. However, once it has gotten some production experience, it may be extended to, say, measure the effect of individual commits to blead on CPAN libraries using the previous monthly development release as a baseline.
This library is currently focused on Perl 5 libraries publicly available on CPAN. In the future, it may be extended to be able to include an organization's private libraries as well.
This library is currently focused on blead, the master branch of the Perl 5 core distribution. However, it could, in principle, be extended to assess the impact on CPAN libraries of code in non-blead ("smoke-me") branches as well.
Currently, if you run code built with this library on a monthly basis, you will produce an updated version of a pipe-separated-values (PSV) plain-text file suitable for opening in a spreadsheet. The columns in that PSV file will be these:
dist perl-5.27.0.author perl-5.27.0.distname perl-5.27.0.distversion perl-5.27.0.grade perl-5.27.1.author perl-5.27.1.distname perl-5.27.1.distversion perl-5.27.1.grade ...
So the output for particular CPAN libraries will look like this:
dist|perl-5.27.0.author|perl-5.27.0.distname|perl-5.27.0.distversion|perl-5.27.0.grade|perl-5.27.1.author|perl-5.27.1.distname|perl-5.27.1.distversion|perl-5.27.1.grade|... Acme-CPANAuthors|ISHIGAKI|Acme-CPANAuthors-0.26|0.26|PASS|ISHIGAKI|Acme-CPANAuthors-0.26|0.26|PASS|... Algorithm-C3|HAARG|Algorithm-C3-0.10|0.10|PASS|HAARG|Algorithm-C3-0.10|0.10|PASS|...
If a particular CPAN library receives a grade of PASS one month and a grade of <FAIL> month, it ought to be inspected for the cause of that breakage. Sometimes the change in Perl 5 is wrong and needs to be reverted. Sometimes the change in Perl 5 is correct (or, at least, plausible) but exposes sub-optimal code in the CPAN module. Sometimes the failure is due to external conditions, such as a change in a C library on the testing platform. There's no way to write code to figure out which situation -- or mix of situations -- we are in. The human user must intervene at this point.
PASS
Platform
The user should select a machine/platform which is likely to be reasonably stable over one Perl 5 annual development cycle. We understand that the platform's system administrator will be updating system libraries for security and other reasons over time. But it would be a hassle to run this software on a machine scheduled for a complete major version update of its operating system.
Perl 5 Configuration
The user must decide on a Perl 5 configuration before using Test-Against-Dev on a regular basis and not change that over the course of the testing period. Otherwise, the results may reflect changes in that configuration rather than changes in Perl 5 core distribution code or changes in the targeted CPAN libraries.
"Perl 5 configuration" means the way one calls Configure when building Perl 5 from source, <e.g.>:
sh ./Configure -des -Dusedevel \ -Duseithreads \ -Doptimize="-O2 -pipe -fstack-protector -fno-strict-aliasing"
So, you should not configure without threads one month but with threads another month. You should not switch to debugging builds half-way through the testing period.
Selection of CPAN Libraries for Testing
This is the most important step in preparation to use this library.
When you use this library, you are in effect saying: Here is a list of CPAN modules important enough to me that I don't want to see them start breaking in the course of Perl's annual development cycle. (If they do break, then the Perl 5 Porters and the modules' authors/maintainers must address how to handle the breakage.) To keep track of the problem, I'm going to build perl from each monthly release and attempt to install this entire list against that perl.
Hence, once you decide to track a certain CPAN library, you should continue to include it in your list of modules to be tracked for the balance of the development cycle. You can, it is true, add additional modules to your list part way through the development cycle. You simply won't have the same baseline data that you have for the modules you selected at the very beginning.
Here are some approaches that come to mind:
CPAN river
The CPAN river is a concept developed by Neil Bowers and other participants in the Perl Toolchain Gang and Perl QA Hackathons and Summits. The concept starts from the premise that CPAN libraries upon which many other CPAN libraries depend are more important than those upon which few other libraries depend. That's a useful definition of importance even if it is not strictly true. Modules "way upstream" feed modules and real-world code "farther downstream". Hence, if Perl 5's development branch changes in a way such that "upstream" modules start to fail to configure, build, test and install correctly, then we have a potentially serious problem. The author of this library has primarily developed it with the idea that it would be run monthly to see what happens with the 1000 "farthest upstream" modules -- the so-called "CPAN River Top 1000".
Organizational dependencies
Many organizations use technologies such as Carton and cpanfile to keep track of their dependencies on CPAN libraries. The lists compile by such applications could very easily be translated into a list of modules tested once a month against a Perl development release.
What repeatedly breaks
Certain CPAN libraries get broken relatively frequently. While this can happen because of sub-standard coding practices in those libraries, it more often happens because these libraries, in order to do what they want to do, reach down deep into the Perl 5 guts and use undocumented or not publicly supported features of Perl.
new()
Purpose
Test::Against::Dev constructor. Guarantees that the top-level directory for the application exists, then creates two directories thereunder: testing/ and results/.
testing/
results/
Arguments
my $self = Test::Against::Dev->new( { application_dir => '/path/to/application', } );
Takes a hash reference with the following elements:
application_dir
String holding path to the directory which will serve as the top level for your application.
Return Value
Test::Against::Dev object.
Comment
This class has two possible constructors: this method and new_from_existing_perl_cpanm(). Use this one when you need to do a fresh install of a perl by compiling it from a downloaded tarball. Use the other one when you have already installed such a perl on disk and have installed a cpanm against that perl.
new_from_existing_perl_cpanm()
The method will guarantee that underneath the application directory there are two directories: testing and results.
perform_tarball_download()
($tarball_path, $work_dir) = $self->perform_tarball_download( { host => 'ftp.funet.fi', hostdir => /pub/languages/perl/CPAN/src/5.0, perl_version => 'perl-5.27.6', compression => 'gz', work_dir => "~/tmp/Downloads", verbose => 1, mock => 0, } );
Hash reference with the following elements:
host
String. The FTP mirror from which you wish to download a tarball of a Perl release. Required.
hostdir
String. The directory on the FTP mirror specified by host in which the tarball is located. Required.
perl_version
String denoting a Perl release. The string must start with perl-, followed by the major version, minor version and patch version delimited by periods. The major version is always 5. Required.
perl-
5
compression
String denoting the compression format of the tarball you wish to download. Eligible compression formats are gz, bz2 and bz2. Required.
gz
bz2
Note that not all compression formats are available for all tarballs on our FTP mirrors and that the compression formats offered may change over time.
Note further that gz is currently the recommended format, as the other methods have not been thorougly tested.
work_dir
String holding absolute path to the directory in which the work of configuring and building the new perl will be performed. Optional; if not provided a temporary directory created via File::Temp::tempdir() will be used.
File::Temp::tempdir()
verbose
Extra information provided on STDOUT. Optional; defaults to being off; provide a Perl-true value to turn it on. Scope is limited to this method.
mock
Display the expected results of the download on STDOUT, but don't actually do it. Optional; defaults to being off; provide a Perl-true value to turn it on. Any program using this option will terminate with a non-zero status once the results have been displayed.
Returns a list of two elements:
Tarball path
String holding absolute path to the tarball once downloaded.
Work directory
String holding path of directory in which work of configuring and building perl will be performed. (This is probably only useful if you want to see the path to the temporary directory. It will be uninitialized if mock is turned on.)
The method guarantees the existence of a directory whose name will be the value of the perl_version argument and which will be found underneath the testing directory (discussed in new() above). This "release directory" -- accessible by calling $self-get_release_dir()> -- will be the directory below which a new perl will be installed.
$self-
configure_build_install_perl()
Configures, builds and installs perl from the downloaded tarball.
my $this_perl = $self->configure_build_install_perl({ verbose => 1, });
configure_command
String holding a shell command to call Perl's Configure program with command-line options. Optional; will default to:
my $release_dir = $self->get_release_dir(); sh ./Configure -des -Dusedevel -Uversiononly -Dprefix=$release_dir \ -Dman1dir=none -Dman3dir=none
The spelling of the command is subsequently accessible by calling $self-access_configure_command()>.
make_install_command
String holding a shell command to build and install perl underneath the release directory. Optional; will default to:
make install
The spelling of the command is subsequently accessible by calling $self-access_make_install_command()>.
Extra information provided on STDOUT. Optional; defaults to being off. Set to 1 (recommended) for moderate verbosity. Set to 2 for extra verbosity (full output of decompression commands, Configure and make). Scope is limited to this method.
1
2
String holding absolute path to the new perl executable. This location can subsequently be accessed by calling $self-get_this_perl()>.
The new perl executable will sit two levels underneath the release directory in a directory named bin/. That directory will sit next to a directory named lib/ under which libraries will be installed. Those locations can subsequently be accessed by calling $self-get_bin_dir()> and $self-get_lib_dir()>, respectively.
fetch_cpanm()
Fetch the fatpacked cpanm executable and install it against the newly installed perl.
my $this_cpanm = $self->fetch_cpanm( { verbose => 1 } );
Hash reference with these elements:
uri
String holding URI from which cpanm will be downloaded. Optional; defaults to http://cpansearch.perl.org/src/MIYAGAWA/App-cpanminus-1.7043/bin/cpanm.
String holding the absolute path to the newly installed cpanm executable.
The executable's location can subsequently be accessed by calling $self-get_this_cpanm()>. The method also guarantees the existence of a .cpanm directory underneath the release directory. This directory can subsequently be accessed by calling $self-get_cpanm_dir()>.
run_cpanm()
Use cpanm to install selected Perl modules against the perl built for testing purposes.
Two mutually exclusive interfaces:
Modules provided in a list
$gzipped_build_log = $self->run_cpanm( { module_list => [ 'DateTime', 'AnyEvent' ], title => 'two-important-libraries', verbose => 1, } );
Modules listed in a file
$gzipped_build_log = $self->run_cpanm( { module_file => '/path/to/cpan-river-file.txt', title => 'cpan-river-1000', verbose => 1, } );
Each interface takes a hash reference with the following elements:
module_list OR module_file
module_list
module_file
Mutually exclusive; must use one or the other but not both.
The value of module_list must be an array reference holding a list of modules for which you wish to track the impact of changes in the Perl 5 core distribution over time. In either case the module names are spelled in Some::Module format -- i.e., double-colons -- rather than in Some-Module format (hyphens).
Some::Module
Some-Module
title
String which will be used to compose the name of project-specific output files. Required.
String holding the absolute path of a gzipped copy of the build.log generated by the cpanm run which this method conducts. The basename of this file, using the arguments supplied, would be:
cpan-river-1000.perl-5.27.6.01.build.log.gz
The method guarantees the existence of several directories underneath the "results" directory discussed above. These are illustrated as follows:
/path/to/application/results/ /results/perl-5.27.6/ /results/perl-5.27.6/analysis/ /results/perl-5.27.6/buildlogs/ /results/perl-5.27.6/storage/
analyze_cpanm_build_logs()
Parse the build.log created by running run_cpanm(), creating JSON files which log the results of attempting to install each module in the list or file.
$ranalysis_dir = $self->analyze_cpanm_build_logs( { verbose => 1 } );
Hash reference which, at the present time, can only take one element: verbose. Optional.
String holding absolute path to the directory holding .log.json files for a particular run of run_cpanm().
analyze_json_logs()
Create a character-delimited-values file summarizing the results of a given run. The delimiter defaults to a pipe (|), thereby creating a pipe-separated values file (.psv), but you may select a comma (,), generating a comma-separated-values file (.csv) as well.
|
.psv
,
.csv
my $fcdvfile = $self->analyze_json_logs( { run => 1, verbose => 1, sep_char => '|' } );
run
A positive integer.
sep_char
Delimiter character. Optional; defaults to pipe (|), but comma (,) may also be chosen.
String holding absolute path to the .psv or .csv file created.
As a precaution, the function creates a tarball to archive the .log.json files for a given run.
Alternate constructor to be used when you have already built a perl executable to be used in tracking Perl development and installed a cpanm against that perl.
perl
cpanm
$self = Test::Against::Dev->new_from_existing_perl_cpanm( { path_to_perl => '/path/to/perl-5.27.0/bin/perl', application_dir => '/path/to/application', perl_version => 'perl-5.27.0', } );
path_to_perl
String holding path to an installed perl executable. Required.
String holding path to the directory which will serve as the top level for your application. (Same meaning as in new().) Required.
String denoting a Perl release. The string must start with perl-, followed by the major version, minor version and patch version delimited by periods. The major version is always 5. (Same meaning as in perform_tarball_download().) Required.
As was the case with new(), this method guarantees the existence of the application directory and the testing and results directories thereunder. It also performs sanity checks for the paths to installed perl and cpanm.
If you already have a perl installed which suffices for a monthly development release, then you can start with this method, omit calls to perform_tarball_download(), configure_build_install_perl() and fetch_cpanm() and go directly to run_cpanm().
This library has a fair number of direct and indirect dependencies on other CPAN libraries. Consequently, the library may experience problems if there are major changes in those libraries. In particular, the code is indirectly dependent upon App::cpanminus::reporter, which in turn is dependent upon cpanm. (Nonetheless, this software could never have been written without those two libraries by Breno G. de Oliveira and Tatsuhiko Miyagawa, respectively.)
James E Keenan CPAN ID: JKEENAN jkeenan@cpan.org http://thenceforward.net/perl
Please report any bugs by mail to bug-Test-Against-Dev@rt.cpan.org or through the web interface at http://rt.cpan.org.
bug-Test-Against-Dev@rt.cpan.org
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
The full text of the license can be found in the LICENSE file included with this module.
Copyright James E Keenan 2017-2018. All rights reserved.
This library emerged in the wake of the author's participation in the Perl 5 Core Hackathon held in Amsterdam, Netherlands, in October 2017. The author thanks the lead organizers of that event, Sawyer X and Todd Rinaldo, for the invitation to the hackathon. The event could not have happened without the generous contributions from the following companies:
Booking.com
cPanel
Craigslist
Bluehost
Assurant
Grant Street Group
perl(1). CPAN::cpanminus::reporter::RetainReports(3). Perl::Download::FTP(3). App::cpanminus::reporter(3). cpanm(3).
2017 Perl 5 Core Hackathon Discussion on Testing.
perl.cpan.testers.discuss Thread on Testing.
To install Test::Against::Dev, copy and paste the appropriate command in to your terminal.
cpanm Test::Against::Dev
CPAN shell
perl -MCPAN -e shell install Test::Against::Dev
For more information on module installation, please visit the detailed CPAN module installation guide.