NAME

Text::TFIDF::Ngram - Compute the TF-IDF measure for ngram phrases

VERSION

version 0.0505

SYNOPSIS

  use Text::TFIDF::Ngram;

  my $t = Text::TFIDF::Ngram->new(
    files => [qw( foo.txt bar.txt )],
    size  => 3,
  );

  my $w = $t->tf( 'foo.txt', 'foo bar baz' );
  my $x = $t->idf('foo bar baz');
  my $y = $t->tfidf( 'foo.txt', 'foo bar baz' );
  printf "TF: %.3f, IDF: %.3f, TF-IDF: %.3f\n", $w, $x, $y;

  my $z = $t->tfidf_by_file;
  print Dumper $z;

DESCRIPTION

This module computes the TF-IDF ("term frequency - inverse document frequency") measure for a corpus of text documents.

This module will only work when given more than one document. Because the idf method is computed based on all documents, a single document in the given corpus will return 0.

ATTRIBUTES

files

ArrayRef of filenames.

size

Integer ngram phrase size.

Default: 1

stopwords

Boolean indicating that phrases with stopwords will be ignored.

Default: 1

punctuation

Regular expression to be used to parse-out unwanted punctuation. Giving the constructor a value of '' or 0 will override this and not exclude any characters from the results.

Default: qr/[-!"#$%&()*+,.\/\\:;<=>?@\[\]^_`{|}~]/

Note that the default does not exclude the single quote.

lowercase

Boolean to render the ngrams in lowercase.

Default: 0

counts

HashRef of the ngram counts of each processed file.

This is a computed attribute. Providing it to the constructor will be ignored.

file_tfidf

HashRef of the TF-IDF values in each processed file.

This is a computed attribute. Providing it to the constructor will be ignored.

METHODS

new

  $t = Text::TFIDF::Ngram->new(
    files       => \@files,
    size        => $size,
    stopwords   => $stopwords,
    punctuation => $punctuation,
    lowercase   => $lowercase,
  );

Create a new Text::TFIDF::Ngram object. If the files argument is passed in, the ngrams of each file are stored in counts.

tf

  $tf = $t->tf( $file, $phrase );

Return the frequency of the given phrase in the document file. This is not the "raw count" of the phrase, but rather the percentage of times it is seen.

idf

  $idf = $t->idf($phrase);

Return the inverse document frequency of a phrase across all corpus documents.

tfidf

  $tfidf = $t->tfidf( $file, $phrase );

Compute the TF-IDF weight for the given file and phrase. If the phrase is not in the corpus, a warning is issued and undef is returned.

tfidf_by_file

  $tfidf = $t->tfidf_by_file;

Construct a HashRef of all files with all phrases and their tfidf values.

SEE ALSO

The eg/* and t/01-methods.t files in this distribution

https://en.wikipedia.org/wiki/Tf%E2%80%93idf

Lingua::EN::Ngram

Lingua::StopWords

List::Util

Moo

AUTHOR

Gene Boggs <gene@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2020 by Gene Boggs.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.