NAME

Text::SpeedyFx - tokenize/hash large amount of strings efficiently

VERSION

version 0.004

SYNOPSIS

    use Data::Dumper;
    use Text::SpeedyFx;

    my $sfx = Text::SpeedyFx->new;

    my $words_bag = $sfx->hash('To be or not to be?');
    print Dumper $words_bag;
    #$VAR1 = {
    #          '1422534433' => '1',
    #          '4120516737' => '2',
    #          '1439817409' => '2',
    #          '3087870273' => '1'
    #        };

    my $feature_vector = $sfx->hash_fv("thats the question", 5);
    print Dumper $feature_vector;
    #$VAR1 = [
    #          '0',
    #          '1',
    #          '0',
    #          '1',
    #          '0'
    #        ];

DESCRIPTION

XS implementation of a very fast combined parser/hasher which works well on a variety of bag-of-word problems.

Original implementation is in Java and was adapted for a better Unicode compliance.

METHODS

new([$seed])

Initialize parser/hasher, optionally using a specified $seed (default: 1).

hash($string)

Parses $string and returns a hash reference where keys are hashed tokens and values are respective count.

hash_fv($string, $n)

Parses $string and returns a feature vector with $n elements.

hash_min($string)

Parses $string and returns the hash with the lowest value.

REFERENCES

• Extremely Fast Text Feature Extraction for Classification and Indexing by George Forman and Evan Kirshenbaum

AUTHOR

Stanislaw Pusep <stas@sysd.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2012 by Stanislaw Pusep.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.