NAME

Lingua::Stem::Snowball - Perl interface to Snowball stemmers.

SYNOPSIS

  use  Lingua::Stem::Snowball;

  my @lang = stemmers();

OO interface:

  my $lang = 'en';
  my $dict = Lingua::Stem::Snowball->new(lang => $lang);
  # Test if $lang is correct
  die $@ if ($@);
  my $locale = 'C'; 

  my $dict = Lingua::Stem::Snowball->new(lang => $lang, locale => $locale);
  my $lemm = $dict->stem($word);
  my $lemm = $dict->stem($word, \$is_stemmed);

  my $dict = Lingua::Stem::Snowball->new();
  $dict->lang($lang);
  $dict->locale($locale);
  my $lemm = $dict->stem($word);
  my @lemm = $dict->stem(\@words);

Plain interface:

  my $lemm = stem($lang, $word);
  my $lemm = stem($lang, $word, $locale);
  my $lemm = stem($lang, $word, $locale, \$is_stemmed);

DESCRIPTION

This module provides unified perl interface to Snowball stemmers (http://snowball.tartarus.org) and virtually supports various languages. It's written using C for high performance and provides OO and plain interfaces.

The motivation of developing this module was to provide a generic access to stemming algorithms for OpenFTS project - full text search engine (http://openfts.sourceforge.net).

The module is very similar with Lingua::Stem. But Lingua::Stem is written in pure perl whereas Lingua::Stem::Snowball is an XS version of the snowball stemmers.

The following stemmers are available (as of Lingua::Stem 0.70):

  |------------------------------|
  | Language     | L:S   | L:S:S | 
  |------------------------------|
  | English      | y     | y     | 
  | French       | y     | y     | 
  | Spanish      | n     | y     | 
  | Portuguese   | y     | y     | 
  | Italian      | y     | y     | 
  | German       | y     | y     | 
  | Dutch        | n     | y     | 
  | Swedish      | y     | y     | 
  | Norwegian    | y     | y     | 
  | Danish       | y     | y     | 
  | Russian      | n     | y     | 
  | Finnish      | n     | y     | 
  | Galician     | y     | n     | 
  |------------------------------|

Here is a little benchmark with examples files from the snowball distribution (with no cache):

  |---------------------------------------------------|
  | Language | Unique |          Time (s)             | 
  |          | words  | L:S:S | L:S:S | L:S   | L:S:S | 
  |          |        | @     | $     | @     | $     | 
  |---------------------------------------------------|
  | DA       | 23829  | 0.5   | 1.1   | 7.3   | 14.2  | 
  | DE       | 35033  | 0.9   | 1.9   | 64.3  | 73.5  | 
  | EN       | 30428  | 0.7   | 1.5   | 2.5   | 8.8   | 
  | FR       | 20403  | 0.6   | 1.1   | 182.7 | 188.0 | 
  | IT       | 35494  | 1.0   | 2.0   | 345.6 | 350.2 | 
  | NO       | 20628  | 0.4   | 1.0   | 14.3  | 20.6  | 
  | PT       | 32016  | 0.8   | 1.7   | 405.6 | 414.8 | 
  | SV       | 30623  | 0.0   | 0.5   | 15.9  | 25.6  | 
  |---------------------------------------------------|

Here is the same benchmark with all unique words found in the bible:

  |---------------------------------------------------|
  | EN       | 12718  | 0.3   | 0.7   | 1.0   | 3.6   | 
  |---------------------------------------------------|

METHODS

$dict = Lingua::Stem::Snowball->new

Creates a new instance of the stemmer.

The constructor takes hash style parameters. The following parameters are recognized:

lang: language (ISO code).

locale: locale.

my $stemmed = $dict->stem($word)

Returns the stemmed word for $word.

my @stemmed = $dict->stem(\@words)

Returns an array of the stemmed words contained in @words.

$dict->lang([$lang])

Accessor for the lang parameter. If there is no stemmer for $lang, the language is not changed.

$dict->locale([$locale])

Accessor for the locale parameter.

stemmers()

Returns a list of all available languages with a stemmer.

$dict->strip_apostrophes([1|0])

By default, the stemmer will not strip apostrophes for you. So, if you make the following call:

  my @words = ('The', 'Ranger\'s', 'Digest');
  my @stemmed = $dict->stem(\@words);

The result might not be what you expected (if you split(' ') a user search entry for example).

Stripping 's in perl can be a little expensive, so you can let the stemmer do it in C:

  my @words = ('The', 'Ranger\'s', 'Digest');
  $dict->strip_apostrophes(1);
  my @stemmed = $dict->stem(\@words);

This method strips 's (english) and l', d', ... (french).

REQUESTS & BUGS

Please report any requests, suggestions or bugs via the RT bug-tracking system at http://rt.cpan.org/ or email to bug-Lingua-Stem-Snowball\@rt.cpan.org.

http://rt.cpan.org/NoAuth/Bugs.html?Dist=Lingua-Stem-Snowball is the RT queue for Lingua::Stem::Snowball. Please check to see if your bug has already been reported.

COPYRIGHT

Copyright 2004-2005

Currently maintained by Fabien Potencier, fabpot@cpan.org Original authors Oleg Bartunov, oleg@sai.msu.su, Teodor Sigaev, teodor@stack.net

This software may be freely copied and distributed under the same terms and conditions as Perl.

Snowball files and stemmers are covered by the BSD license.

SEE ALSO

http://snowball.tartarus.org, Lingua::Stem