 NAME
 SYNOPSIS
 DESCRIPTION
 METHODS
 OPTIONS
 EXAMPLE
 REFERENCES
 SEE ALSO
 TO DO/BUGS
 AUTHOR/LICENSE
 DISCLAIMER
 END
NAME
Statistics::Sequences::Vnomes  The Serial Test (psisquare) and Generalized Serial Test (delta psisquare) for equiprobability of vnomes (or vplets/bits) (Good's and KendallBabington Smith's tests)
SYNOPSIS
use Statistics::Sequences::Vnomes 0.11;
my $vnomes = Statistics::Sequences::Vnomes>new();
$vnomes>load(qw/1 0 0 0 1 1 0 1 1 0 0 1 0 0 1 1 1 1 0 1/); # data treated categorically  could also be, e.g., 'a' and 'b'
my $r_freq_href = $vnomes>observed(length => 3); # returns hashref of frequency distribution for trinomes in the sequence
my @freq = $vnomes>observed(length => 3); # returns only the observed trinome frequencies (not keyed by the trinomes themselves)
$val = $vnomes>expected(length => 3); # mean chance expectation for the frequencies (2.5)
$val = $vnomes>psisq(length => 3); # Good's "second backward differences" psisquare (3.4); option 'delta' gives alternative estimates
$val = $vnomes>p_value(length => 3, tails => 1); # 1tailed pvalue for psisquare (0.0913)
$val = $vnomes>z_value(length => 3, tails => 1, ccorr => 1); # inversephi of the 1tailed pvalue (1.333)
my $href = $vnomes>stats_hash(length => 3, values => {psisq => 1, p_value => 1}, tails => 1); # include any statmethod (& their options)
$vnomes>dump(length => 3,
values => {observed => 1, expected => 1, psisq => 1, p_value => 1}, # what stats to show (or not if => 0)
format => 'table', flag => 1, precision_s => 3, precision_p => 7, verbose => 1, tails => 1);
# prints:
# Vnomes (3) statistics
#.+++.
# observed  expected  p_value  psisq 
#    
#+++++
# '011' = 4  2.500  0.0913418  3.400 
# '010' = 1    
# '111' = 2    
# '000' = 1    
# '101' = 2    
# '001' = 3    
# '100' = 3    
# '110' = 4    
#'+++'
#psisq is Good's delta^2psi^2, calculated with second backward differences, and has 2 degrees of freedom.
#pvalue is 1tailed.
# these and other methods inherited from Statistics::Sequences and its parent, Statistics::Data:
$vnomes>dump_data(delim => ','); # commaseparated single line of the loaded data
$vnomes>save(path => 'seq.csv'); # serialized for retrieval by open() method
DESCRIPTION
Implements tests of the independence of successive elements of a sequence of data: serial tests for vnomes (a.k.a vplets or, for binary data, vbits)  singlets/monobits, dinomes/doublets, trinomes/triplets, etc.. Test are of variations in subsequence length v that are equally likely for the sampled sequence. For example, a sequence sampled from a "heads'n'tails" (H and T) distribution can be tested for its equal representation of the trinomes HTH, HTT, TTT, THT, and so on. Counting up these vnomes at all points in the sequence, permitting overlaps, yields a statistic  psisquare  that is approximately distributed as chisquare; the KendallBabington Smith statistic.
However, because these counts are not independent (given the overlaps), Good's Generalized Serial Test is the default teststatistic returned by this module's test routine: It computes psisquare by differencing, viz., in relation to not only the specified length, or value of v, but also its value for the first two prior lengths of v, yielding a statistic, deltasquarepsisquare (the "second backward difference" measure) that is exactly distributed as chisquare.
The test is suitable for multistate data, not only the binary, dichotomous sequence suitable for the runs or joins tests. It can also be used to test that the individual elements in the list are uniformly distributed, that the states are equally represented, i.e., as a chisquarebased frequency test (a.k.a. test of uniformity, equiprobability, equidistribution). This is done by giving a length of 1, i.e., testing for mononomes.
Note that this is not the socalled serial test described by Knuth (1998, Ch. 2), which involves nonoverlapping pairs of sequences.
METHODS
Methods include those described in Statistics::Sequences, which can be used directly from this module, as follows.
new
$vnomes = Statistics::Sequences::Vnomes>new();
Returns a new Vnomes object. Expects/accepts no arguments but the classname.
load
$vnomes>load(@data); # anonymously
$vnomes>load(\@data);
$vnomes>load('sample1' => \@data); # labelled whatever
Loads data anonymously or by name  see load in Statistics::Data for ways data can be loaded and retrieved (more than shown here). Every load unloads all previous loads and any additions to them.
Data for this test of sequences can be categorical or numerical, all being treated categorically. Also, the data do not have to be dichotomous (unlike in tests of runs and joins.
add, read, unload
See Statistics::Data for these and other operations on loaded data.
observed, vnomes_observed, vno
$href = $vnomes>observed(length => int, circularize => 10); # returns keyed distribution; assumes data have already been loaded
@ari = $vnomes>observed(data => [1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1], length => int, circularize => 10); # returns frequencies only
Returns the frequency distribution for the observed number of vnomes of the given length in a sequence. These are counted up as overlapping. Called in array context, returns an array of the counts for each possible pattern; otherwise, where these are keyed by the pattern in a hash reference. For descriptives of the observed frequencies, try calling Statistics::Lite methods with this array.
A value for length greater than zero is required, and must be no more than the samplesize. So for the sequence
1 0 0 0 1 0 0 1 0 1 1 0
there are 12 vnomes of length 1 (mononomes, the number of elements) from the two (0 and 1) that are possible; 11 dinomes from the four (10, 00, 00, 01) that are possible; and 10 trinomes from eight (100, 000, 001, 010, etc.) that are possible.
By default, the sequence is counted up for vnomes as a cyclic sequence, treating the first element of the sequence as following the last one. So the count loops to the beginning of the sequence until all elements from the end are included, and instead of ending the count for trinomes in this sequence at '110', the count includes '101' and '010', increasing the observed sum of trinomes to 12. Set circularize => 0 if the count is not to be made cyclically.
The data to test can already have been loaded, or you send it directly keyed as data.
In the code for this and/or other methods, following the work of Good (1953, 1957; Good & Gover, 1967), v is used to define variables referring to the length of the subsequence (mononome, dinome, trinome, etc.), t defines the number of states (events, letters, etc.) in the sequence, and r identifies each possible subsequence of length v.
expected, vnomes_expected, vne
$count = $vnomes>expected(length => int, circularize => 10); # assumes data have already been loaded
$count = $vnomes>expected(data => [1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1], length => int, circularize => 10, states => [0, 1]);
Returns the expected number of observations for each vnome of the given length in a sequence; i.e., the mean chance expectation, assuming that each event is generated by a random uniform process. Options are as for observed. This expected frequency is given by:
E[V] = Nt^{–v}
where t is the number of possible states (alternatives; as read from the data or as explicitly given as an array in states), v is the vnome length, and N is the length of the sequence, less v + 1 if the count is not to be circularized (Good, 1953, Eq. 12).
Another way to think of this is as the number of mononome observations in the sequence divided by the number of possible permutations of its states for the given length. So, for a sequence made up of 0s and 1s, there are four possible variations of length 2 (00, 10, 01 and 11), so that the expected frequency for each of these variations in a sequence of 20 values is 20 / 4, i.e., 5.
variance
$var = $vnomes>variance(length => int, circularize => 10); # assumes data have already been loaded
$var = $vnomes>variance(data => [1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1], length => int, circularize => 10, states => [0, 1]);
Returns the variance in the expected frequency of each vnome, as per Good (1953, Eqs. 11 and 14).
stdev, standard_deviation
$var = $vnomes>stdev(length => int, circularize => 10); # assumes data have already been loaded
$var = $vnomes>stdev(data => [1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 1], length => int, circularize => 10, states => [0, 1]);
Returns the standard deviation in the expected frequency of each vnome, as per Good (1953, Eqs. 11 and 14).
psisq
$vnomes>psisq(length => int, delta => '210', circularize => '10', states => [qw/A C G T/]);
Performs Good's Generalized Serial Test (by default), of vnomes on the given or named distribution, yielding a psisquare statistic. The option delta specifies if backward differencing.
 delta => 0

The raw psisquare value for subsequences of length v (KendallBabington Smith statistic), i.e., without backward differencing.
Ψ^{2}_{v} = ∑{r} ( n_{r}– (N – v + 1) / t^{v} )^{2} ) / ( (N – v + 1) / t^{v} )
for uncircularized sequences (Good, 1953, Eq. 1), and
Ψ^{2}_{v} = ∑{r} ( n_{r}– Nt^{–v} ) / Nt^{–v}
for circularized sequences (Good, 1953, Eq. 2), where an overline indicates circularization, and where N is the length of the sequence, v is the vnome length, t is the number of unique states that each element of the sequence can take, and r is the number of variations of length v for the given number of unique states.
This statistic is only asymptotically distributed as chisquare if length (v) = 1, and is therefore not used as the default.
 delta => 1

The "first backward differences" of psisquare, which is the difference between the psisquare values for subsequences of length v and length v  1.
ΔΨ^{2}_{v} = Ψ^{2}_{v} – Ψ^{2}_{v–1} (v ≥ 1)
While it is chisquare distributed, counts of firstdifferences are not statistically independent (Good, 1953; Good & Gover, 1967), and "the sequence of second differences forms a much better set of statistics for testing the hypothesis of flatrandomness" (Good & Gover, 1967, p. 104).
 delta => 2 (or undef)

This method returns by default (without specifying a value for delta), or if delta is not 1 or 0, the "second backward differences" psisquare (delta^2psi^2). This incorporates psisquare values for backwardly adjacent values of length (v), i.e., for subsequences of length v, v  1, and v  2.
Δ^{2}Ψ^{2}_{v} = Ψ^{2}_{v} – 2Ψ^{2}_{v–1} – Ψ^{2}_{v–2} (v ≥ 2)
It is not only asymptotically chisquare distributed, but uses statistically independent counts of all the possible variations of sequences of the given length (Good, 1953).
See also Good's algorithm in individual papers describing application of the Serial Test (e.g., Davis & Akers, 1974).
z_value, vnomes_zscore, vzs, zscore
$val = $vnomes>z_value(); # data already loaded, use default windows and prob
$val = $vnomes>z_value(data => $aref);
($zvalue, $pvalue) = $vnomes>z_value(data => $aref, tails => 2); # same but wanting an array, get the pvalue too
Returns the zscore not from a direct test of deviation but from the (inversephi) of the p_value, using Math::Cephes ndtri
. Called in array context, also returns the pvalue itself. Same options and conditions as above. Other options are precision_s (for the z_value) and precision_p (for the p_value).
p_value, test, vnomes_test, vnt
$p = $vnomes>p_value(); # using loaded data and default args
$p = $vnomes>p_value(data => [1, 0, 1, 1, 0], exact => 1); # using given data (bypassing load and read)
$p = $vnomes>p_value(trials => 20, observed => 10); # without using data
Returns probability of obtaining the psisq value for data already loaded, or directly keyed as data. The pvalue is read off the complemented chisquare distribution (incomplete gamma integral) using Math::Cephes igamc
.
dump
$vnomes>dump(length => 3, values => {psisq => 1, p_value => 1}, format => 'tablelablinecsv', flag => 1, precision_s => 3, precision_p => 7, verbose => 1, tails => 1);
Print Vnometest results to STDOUT. See dump in the Statistics::Sequences manpage for details. If verbose => 1, then you get (1) the actual teststatistic depending on the value of delta tested (delta^2psi^2 for the second difference measure (default), deltapsi^2 for the first difference measure, and psi2 for the raw measure), followed by degreesoffreedom in parentheses; and (2) a warning, if relevant, that your length value might be too large with respect to the sample size (see NIST reference, above, in discussing length). If text => 1, you just get the average observed and expected frequencies for each vnome, the Zvalue, and its associated pvalue.
nnomes
$r = $vnomes>nnomes(length => int, states => \@ari); # minimal option required; the "v" value itself, with number of states
$r = $vnomes>nnomes(length => int); # minimal option required, assuming data are loaded so can read its number of states
$r = $vnomes>nnomes(length => int, data=> \@data); # minimal option required; the "v" value itself, with these data
Returns the number of possible subsequences of the given length (v) for the given number of states (t). This is quantity denoted as r in Good's (1953, 1957) papers; i.e.,
r[V] = t^{v}
The routine needs to know two things: the "v" value itself, i.e., the length of the possible subsequences to test (mononomes, dinomes, trinomes, etc.), and the number of states (events, letters, etc.) that the process generating the data could take (from 1 to whatever). The former is always required to be specified by the named argument length. The latter (the number of states) can be directly taken from the named argument nstates, indirectly from the size of the array referenced to the named argument states, from whatever states exist in the array referenced to the named argument data, or from whatever states exist in data already loaded.
prob_r
$Pr = $vnomes>prob_r(length => $v); # length is 1 (mononomes) by default
Returns the probability of the occurrence of any of the individual elements ("digits") in the sequence (v = 1), or of the given length, assuming they are equally likely and independent.
P_{r} = t^{–v}
OPTIONS
Options common to the above stats methods.
length
This is currently a required "option", giving the length of the vnome of interest, i.e., the value of v  an integer greater than or equal to 1, and smaller than than the samplesize.
What is a meaningful maximal value of length? As a chisquare test, it is conventionally required that the expected frequency is at least 5 for each vnome (Knuth, 1988). This can be judged to be too conservative (Delucchi, 1993). The NIST documentation on the serial test (Rukhin et al., 2010) recommends that length should be less than the floored value of log2 of the samplesize, minus 2. No tests are here made of these recommendations.
circularize
By default, observed and expected counts, and the value of psisq, are made by treating the sequence as a cyclic one, where the first element of the sequence follows the last one. This affects (and simplifies) the calculation of the expected frequency of each vnome, and so the value of each psisquare. Also, circularizing ensures that the expected frequencies are accurate; otherwise, they might only be approximate. As Good and Gover (1967) state, "It is convenient to circularize in order to get exact checks of the arithmetic and also in order to simplify some of the theoretical formulae" (p. 103). These methods, however, can also treat the sequence noncyclically by calling them with circularize => 0.
states
Optionally send a referenced array listing the unique states (or 'events', 'letters') in the population from which the sequence was sampled, e.g., states => [qw/A C G T/]. This is useful if the sequence itself might not include all the possible states. If this is not specified, the states are identified from the sequence itself. If giving a list of states, a check in each test is made to ensure that the sequence contains only those elements in the list.
EXAMPLE
Seating at the diner
This is the data from Swed and Eisenhart (1943) also given as an example for the Runs test and Turns test. It lists the occupied (O) and empty (E) seats in a row at a lunch counter. Have people taken up their seats on a random basis  or do they show some social phobia, or are they trying to pick up? What does the test of Vnomes reveal?
use Statistics::Sequences::Vnomes;
my $vnomes = Statistics::Sequences::Vnomes>new();
my @seating = (qw/E O E E O E E E O E E E O E O E/);
$vnomes>load(\@seating);
$vnomes>dump(length => 3, values => {z_value => 1, p_value => 1}, format => 'labline', flag => 1, precision_s => 3, precision_p => 3, tails => 1);
This prints:
z_value = 2.015, p_value = 0.022*
That is, the observed frequency of each possible trio of seating arrangements (the trinomes OOO, OOE, OEE, EEE, etc.) differed significantly from that expected. Look up the observed frequencies for each possible trinome to see if this is because there are more empty or occupied neighbouring seats ("phobia" or "philia"):
$vnomes>dump(length => 3, values => {observed => 1}, format => 'labline');
This prints:
observed = ('OEE' = 4,'EEO' = 4,'EEE' = 2,'OEO' = 1,'EOE' = 5,'OOO' = 0,'OOE' = 0,'EOO' = 0)
As the chanceexpected frequency is 2.5 (from the expected method), there are clearly more than expected trinomes involving empty seats than occupied seats  suggesting a nonrandom factor like social phobia (or body odour?) is at work in sequencing people's seating here. Noting that the sequencing isn't significant for dinomes (with length => 2) might also tell us something about what's going on. What happens for vnomes of 4 or more in length? Maybe the runs or pot test might be a better summary of what's going on.
REFERENCES
Davis, J. W., & Akers, C. (1974). Randomization and tests for randomness. Journal of Parapsychology, 38, 393407.
Delucchi, K. L. (1993). The use and misuse of chisquare: Lewis and Burke revisited. Psychological Bulletin, 94, 166176.
Gatlin, L. L. (1979). A new measure of bias in finite sequences with applications to ESP data. Journal of the American Society for Psychical Research, 73, 2943. (Used for one of the reference tests in the CPAN distribution.)
Good, I. J. (1953). The serial test for sampling numbers and other tests for randomness. Mathematical Proceedings of the Cambridge Philosophical Society, 49, 276284.
Good, I. J. (1957). On the serial test for random sequences. Annals of Mathematical Statistics, 28, 262264.
Good, I. J., & Gover, T. N. (1967). The generalized serial test and the binary expansion of [squareroot]2. Journal of the Royal Statistical Society A, 130, 102107.
Kendall, M. G., & Babington Smith, B. (1938). Randomness and random sampling numbers. Journal of the Royal Statistical Society, 101, 147166.
Knuth, D. E. (1998). The art of computer programming (3rd ed., Vol. 2 Seminumerical algorithms). Reading, MA, US: AddisonWesley.
Rukhin, A., Soto, J., Nechvatal, J., Smid, M., Barker, E., Leigh, S., et al. (2010). A statistical test suite for random and pseudorandom number generators for cryptographic applications. Retrieved September 4 2010, from http://csrc.nist.gov/groups/ST/toolkit/rng/documents/SP80022b.pdf, and July 17, 2013, from http://csrc.nist.gov/publications/nistpubs/80022rev1a/SP80022rev1a.pdfSP80022rev1a.pdf (revised).
SEE ALSO
Statistics::Sequences submodules for other tests of sequences, and for sharing data between these tests.
TO DO/BUGS
Handle nonoverlapping vnomes.
AUTHOR/LICENSE
 Copyright (c) 20062013 Roderick Garton

rgarton AT cpan DOT org
This program is free software. It may be used, redistributed and/or modified under the same terms as Perl5.6.1 (or later) (see http://www.perl.com/perl/misc/Artistic.html).
DISCLAIMER
To the maximum extent permitted by applicable law, the author of this module disclaims all warranties, either express or implied, including but not limited to implied warranties of merchantability and fitness for a particular purpose, with regard to the software and the accompanying documentation.
END
This ends documentation of the Perl implementation of the psisquare statistic, KendallBabington Smith test, and Good's Generalized Serial Test, for randomness in a sequence.