Author image James E Keenan
and 1 contributors

NAME

Devel::Git::MultiBisect::AllCommits - Gather test output over an entire range of git commits

SYNOPSIS

    use Devel::Git::MultiBisect::AllCommits;

    $self = Devel::Git::MultiBisect::AllCommits->new(\%parameters);

    $commit_range = $self->get_commits_range();

    $full_targets = $self->set_targets(\@target_args);

    $outputs = $self->run_test_files_on_one_commit($commit_range->[0]);

    $all_outputs = $self->run_test_files_on_all_commits();

    $rv = $self->get_digests_by_file_and_commit();

    $transitions = $self->examine_transitions();

DESCRIPTION

Given a Perl library or application kept in git for version control, it is often useful to be able to compare the output collected from running one or several test files over a range of git commits. If that range is sufficiently large, a test may fail in more than one way over that range.

If that is the case, then simply asking, "When did this file start to fail?" is insufficient. We may want to capture the test output for each commit, or, more usefully, may want to capture the test output only at those commits where the output changed.

Devel::Git::MultiBisect provides methods to achieve that objective. More specifically:

  • When you want to capture the test output for each commit in a specified range, you can use this package, Devel::Git::MultiBisect::AllCommits.

  • When the number of commits in the specified range is large and you only need the test output at those commits where the output materially changed, you can use another package found in this library, Devel::Git::MultiBisect::Transitions.

METHODS

This package inherits methods from Devel::Git::MultiBisect. Only methods unique to Devel::Git::MultiBisect::AllCommits are documented here. See the documentation for Devel::Git::MultiBisect for all other methods, including:

    new()
    get_commits_range()
    set_targets()
    run_test_files_on_one_commit()

run_test_files_on_all_commits()

  • Purpose

    Capture the output from a run of the selected test files at each specific git checkout in the selected commit range.

  • Arguments

        $all_outputs = $self->run_test_files_on_all_commits();

    None; all data needed is already present in the object.

  • Return Value

    Array reference, each of whose elements is an array reference, each of whose elements is a hash reference with the same four keys as in the return value from run_test_files_on_one_commit():

        commit
        commit_short
        file
        file_stub
        md5_hex

    Example:

        [
          # Array where each element corresponds to a single git checkout
    
          [
            # Array where each element corresponds to one of the selected test
            # files (here, 2 test files were targetd)
    
            {
              # Hash where each element correponds to the result of running a
              # single test file at a single commit point
    
              commit => "2a2e54af709f17cc6186b42840549c46478b6467",
              commit_short => "2a2e54a",
              file => "/tmp/AHC_YkwUYg/2a2e54a.t_44_func_hashes_mult_unsorted_t.output.txt",
              file_stub => "t_44_func_hashes_mult_unsorted_t",
              md5_hex => "31b7c93474e15a16d702da31989ab565",
            },
            {
              commit => "2a2e54af709f17cc6186b42840549c46478b6467",
              commit_short => "2a2e54a",
              file => "/tmp/AHC_YkwUYg/2a2e54a.t_45_func_hashes_alt_dual_sorted_t.output.txt",
              file_stub => "t_45_func_hashes_alt_dual_sorted_t",
              md5_hex => "6ee767b9d2838e4bbe83be0749b841c1",
            },
          ],
          [
            {
              commit => "a624024294a56964eca53ec4617a58a138e91568",
              commit_short => "a624024",
              file => "/tmp/AHC_YkwUYg/a624024.t_44_func_hashes_mult_unsorted_t.output.txt",
              file_stub => "t_44_func_hashes_mult_unsorted_t",
              md5_hex => "31b7c93474e15a16d702da31989ab565",
            },
            {
              commit => "a624024294a56964eca53ec4617a58a138e91568",
              commit_short => "a624024",
              file => "/tmp/AHC_YkwUYg/a624024.t_45_func_hashes_alt_dual_sorted_t.output.txt",
              file_stub => "t_45_func_hashes_alt_dual_sorted_t",
              md5_hex => "6ee767b9d2838e4bbe83be0749b841c1",
            },
          ],
        # ...
          [
            {
              commit => "d304a207329e6bd7e62354df4f561d9a7ce1c8c2",
              commit_short => "d304a20",
              file => "/tmp/AHC_YkwUYg/d304a20.t_44_func_hashes_mult_unsorted_t.output.txt",
              file_stub => "t_44_func_hashes_mult_unsorted_t",
              md5_hex => "31b7c93474e15a16d702da31989ab565",
            },
            {
              commit => "d304a207329e6bd7e62354df4f561d9a7ce1c8c2",
              commit_short => "d304a20",
              file => "/tmp/AHC_YkwUYg/d304a20.t_45_func_hashes_alt_dual_sorted_t.output.txt",
              file_stub => "t_45_func_hashes_alt_dual_sorted_t",
              md5_hex => "6ee767b9d2838e4bbe83be0749b841c1",
            },
          ],
        ]
  • Comment

    Note: If the number of commits in the commits range is large, this method will take a long time to run. That time will be even longer if the configuration and build times for each commit are large. For example, to run one test over 160 commits from the Perl 5 core distribution might take 15 hours. YMMV. If either of these conditions holds, you are probably better off using Devel::Git::MultiBisect::Transitions.

    The implementation of this method is very much subject to change.

get_digests_by_file_and_commit()

  • Purpose

    Present the same outcomes as run_test_files_on_all_commits(), but formatted by target file, then commit.

  • Arguments

        $rv = $self->get_digests_by_file_and_commit();

    None; all data needed is already present in the object.

  • Return Value

    Reference to a hash keyed on the basename of the target file, modified to substitute underscores for forward slashes and dots. The value of each element in the hash is a reference to an array which, in turn, holds a list of hash references, one per git commit. Each such hash has the following keys:

        commit
        file
        md5_hex

    Example:

        {
          t_44_func_hashes_mult_unsorted_t   => [
              {
                commit  => "2a2e54af709f17cc6186b42840549c46478b6467",
                file    => "/tmp/Xhilc8ZSgS/2a2e54a.t_44_func_hashes_mult_unsorted_t.output.txt",
                md5_hex => "31b7c93474e15a16d702da31989ab565",
              },
              {
                commit  => "a624024294a56964eca53ec4617a58a138e91568",
                file    => "/tmp/Xhilc8ZSgS/a624024.t_44_func_hashes_mult_unsorted_t.output.txt",
                md5_hex => "31b7c93474e15a16d702da31989ab565",
              },
              # ...
              {
                commit  => "d304a207329e6bd7e62354df4f561d9a7ce1c8c2",
                file    => "/tmp/Xhilc8ZSgS/d304a20.t_44_func_hashes_mult_unsorted_t.output.txt",
                md5_hex => "31b7c93474e15a16d702da31989ab565",
              },
          ],
          t_45_func_hashes_alt_dual_sorted_t => [
              {
                commit  => "2a2e54af709f17cc6186b42840549c46478b6467",
                file    => "/tmp/Xhilc8ZSgS/2a2e54a.t_45_func_hashes_alt_dual_sorted_t.output.txt",
                md5_hex => "6ee767b9d2838e4bbe83be0749b841c1",
              },
              {
                commit  => "a624024294a56964eca53ec4617a58a138e91568",
                file    => "/tmp/Xhilc8ZSgS/a624024.t_45_func_hashes_alt_dual_sorted_t.output.txt",
                md5_hex => "6ee767b9d2838e4bbe83be0749b841c1",
              },
              # ...
              {
                commit  => "d304a207329e6bd7e62354df4f561d9a7ce1c8c2",
                file    => "/tmp/Xhilc8ZSgS/d304a20.t_45_func_hashes_alt_dual_sorted_t.output.txt",
                md5_hex => "6ee767b9d2838e4bbe83be0749b841c1",
              },
          ],
        }
  • Comment

    This method currently presumes that you have called run_test_files_on_all_commits(). It will die otherwise.

examine_transitions()

  • Purpose

    Determine whether a run of the same targeted test file run at two consecutive commits produced the same or different output (as measured by string equality or inequality of each commit's md5_hex value.

  • Arguments

        $hashref = $self->get_digests_by_file_and_commit();
    
        $transitions = $self->examine_transitions($hashref);

    Hash reference returned by get_digests_by_file_and_commit();

  • Return Value

    Reference to a hash keyed on the basename of the target file, modified to substitute underscores for forward slashes and dots. The value of each element in the hash is a reference to an array which, in turn, holds a list of hash references, one per each pair of consecutive git commits. Each such hash has the following keys:

        older
        newer
        compare

    The value for each of the older and newer elements is a reference to a hash with two elements:

        md5_hex
        idx

    ... where md5_hex is the digest of the test output file and idx is the position (count starting at 0) of that element in the list of commits in the commit range.

    Example:

        {
          t_44_func_hashes_mult_unsorted_t   => [
              {
                compare => "same",
                newer   => { md5_hex => "31b7c93474e15a16d702da31989ab565", idx => 1 },
                older   => { md5_hex => "31b7c93474e15a16d702da31989ab565", idx => 0 },
              },
              {
                compare => "same",
                newer   => { md5_hex => "31b7c93474e15a16d702da31989ab565", idx => 2 },
                older   => { md5_hex => "31b7c93474e15a16d702da31989ab565", idx => 1 },
              },
              # ...
              {
                compare => "same",
                newer   => { md5_hex => "31b7c93474e15a16d702da31989ab565", idx => 9 },
                older   => { md5_hex => "31b7c93474e15a16d702da31989ab565", idx => 8 },
              },
          ],
          t_45_func_hashes_alt_dual_sorted_t => [
              {
                compare => "same",
                newer   => { md5_hex => "6ee767b9d2838e4bbe83be0749b841c1", idx => 1 },
                older   => { md5_hex => "6ee767b9d2838e4bbe83be0749b841c1", idx => 0 },
              },
              {
                compare => "same",
                newer   => { md5_hex => "6ee767b9d2838e4bbe83be0749b841c1", idx => 2 },
                older   => { md5_hex => "6ee767b9d2838e4bbe83be0749b841c1", idx => 1 },
              },
              {
                compare => "same",
                newer   => { md5_hex => "6ee767b9d2838e4bbe83be0749b841c1", idx => 3 },
                older   => { md5_hex => "6ee767b9d2838e4bbe83be0749b841c1", idx => 2 },
              },
              # ...
              {
                compare => "same",
                newer   => { md5_hex => "6ee767b9d2838e4bbe83be0749b841c1", idx => 9 },
                older   => { md5_hex => "6ee767b9d2838e4bbe83be0749b841c1", idx => 8 },
              },
          ],
        }
  • Comment

    This method currently may be called only after calling run_test_files_on_all_commits() and will die otherwise.

    Since in this method we are concerned with the transition in the test output between a pair of commits, the second-level arrays returned by this method will have one fewer element than the second-level arrays returned by get_digests_by_file_and_commit().