Ted Pedersen




Statistical library package to calculate the T-score. This package should be used with statistic.pl and rank.pl.


Assume that the frequency count data associated with a bigram <word1><word2> is stored in a 2x2 contingency table:

          word2   ~word2
  word1    n11      n12 | n1p
 ~word1    n21      n22 | n2p
           np1      np2   npp

where n11 is the number of times <word1><word2> occur together, and n12 is the number of times <word1> occurs with some word other than word2, and n1p is the number of times in total that word1 occurs as the first word in a bigram.

The T-score is defined as a ratio of difference between the observed and the expected mean to the variance of the sample. Note that this is a variant of the standard t-test that was proposed for use in the identification of collocations in large samples of text.

Thus, the T-score is defined as follows:

    m11 = n1p * np1 / npp
    T-score = (n11 - m11)/sqrt(n11)


Ted Pedersen <tpederse@d.umn.edu>

Satanjeev Banerjee <banerjee@cs.cmu.edu>

Bridget Thomson McInnes <bthomson@d.umn.edu>


This measure currently only defined for bigram data stored in 2x2 contingency table.


Mailing List: http://groups.yahoo.com/ngram


Copyright 2000-2004 by Ted Pedersen and Satanjeev Banerjee

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.