The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.
 / 
Plucene-1.25
River stage two • 6 direct dependents • 19 total dependents
/ Plucene::Analysis::LetterTokenizer

NAME

Plucene::Analysis::LetterTokenizer - Letter tokenizer

SYNOPSIS

        # isa Plucene::Analysis::CharTokenizer

DESCRIPTION

This is the letter tokenizer class, which divides text at non-letters.

Note: this does a decent job for most European languages, but does a terrible job for some Asian languages, where words are not separated by spaces