NAME

Search::Fulltext::Tokenizer::Ngram - Character n-gram tokenizer for Search::Fulltext

VERSION

version 0.01

SYNOPSIS

  use utf8;
  use Search::Fulltext;
  use Search::Fulltext::Tokenizer::Bigramm;
  
  my $searcher = Search::Fulltext->new(
      docs => [
          'ハンプティ・ダンプティ 塀の上',
          'ハンプティ・ダンプティ 落っこちた',
          '王様の馬みんなと 王様の家来みんなでも',
          'ハンプティを元に 戻せなかった',
      ],
      tokenizer => q/perl 'Search::Fulltext::Tokenizer::Bigram::get_tokenizer'/,
  );
  my $hit_document_ids = $searcher->search('ハンプティ');  # [0, 1, 3]

DESCRIPTION

This module provides character N-gram tokenizers for Search::Fulltext.

By default {1,2,3}-gram tokenzers are available.

CREATING A N(> 3)-GRAM TOKENIZER

If you wish to use other N-grams where N > 3, you can create it by inheriting Search::Fulltext::Tokenizer::Ngram:

  package My::Tokenizer::42gram;
  
  use parent qw/Search::Fulltext::Tokenizer::Ngram/;
  
  my $iterator_generator = __PACKAGE__->new(42);
  
  sub get_tokenizer {
      sub { $iterator_generator->create_token_iterator(@_) };
  }

SEE ALSO

Search::Fulltext::Tokenizer::Unigram Search::Fulltext::Tokenizer::Bigram Search::Fulltext::Tokenizer::Trigram

AUTHOR

Koichi SATOH <sekia@cpan.org>

COPYRIGHT AND LICENSE

This software is Copyright (c) 2014 by Koichi SATOH.

This is free software, licensed under:

  The MIT (X11) License