KinoSearch::Docs::Tutorial::Analysis - How to choose and use Analyzers.
Try swapping out the PolyAnalyzer in our Schema for a Tokenizer:
my $tokenizer = KinoSearch::Analysis::Tokenizer->new; my $type = KinoSearch::FieldType::FullTextType->new( analyzer => $tokenizer, );
Search for senate, Senate, and Senator before and after making the change and re-indexing.
senate
Senate
Senator
Under PolyAnalyzer, the results are identical for all three searches, but under Tokenizer, searches are case-sensitive, and the result sets for Senate and Senator are distinct.
What's happening is that PolyAnalyzer is performing more aggressive processing than Tokenizer. In addition to tokenizing, it's also converting all text to lower case so that searches are case-insensitive, and using a "stemming" algorithm to reduce related words to a common stem (senat, in this case).
senat
PolyAnalyzer is actually multiple Analyzers wrapped up in a single package. In this case, it's three-in-one, since specifying a PolyAnalyzer with language => 'en' is equivalent to this snippet:
language => 'en'
my $case_folder = KinoSearch::Analysis::CaseFolder->new; my $tokenizer = KinoSearch::Analysis::Tokenizer->new; my $stemmer = KinoSearch::Analysis::Stemmer->new( language => 'en' ); my $polyanalyzer = KinoSearch::Analysis::PolyAnalyzer->new( analyzers => [ $case_folder, $tokenizer, $stemmer ], );
You can add or subtract Analyzers from there if you like. Try adding a fourth Analyzer, a Stopalizer for suppressing "stopwords" like "the", "if", and "maybe".
my $stopalizer = KinoSearch::Analysis::Stopalizer->new( language => 'en', ); my $polyanalyzer = KinoSearch::Analysis::PolyAnalyzer->new( analyzers => [ $case_folder, $tokenizer, $stopalizer, $stemmer ], );
Also, try removing the Stemmer.
my $polyanalyzer = KinoSearch::Analysis::PolyAnalyzer->new( analyzers => [ $case_folder, $tokenizer ], );
The original choice of a stock English PolyAnalyzer probably still yields the best results for this document collection, but you get the idea: sometimes you want a different Analyzer.
Sometimes you don't want an Analyzer at all. That was true for our "url" field because we didn't need it to be searchable, but it's also true for certain types of searchable fields. For instance, "category" fields are often set up to match exactly or not at all, as are fields like "last_name" (because you may not want to conflate results for "Humphrey" and "Humphries").
To specify that there should be no analysis performed at all, use StringType:
my $type = KinoSearch::FieldType::StringType->new; $schema->spec_field( name => 'category', type => $type );
In our next tutorial chapter, KinoSearch::Docs::Tutorial::Highlighter, we'll add highlighted excerpts from the "content" field to our search results.
Copyright 2008-2009 Marvin Humphrey
See KinoSearch version 0.30.
To install KSx::Simple, copy and paste the appropriate command in to your terminal.
cpanm
cpanm KSx::Simple
CPAN shell
perl -MCPAN -e shell install KSx::Simple
For more information on module installation, please visit the detailed CPAN module installation guide.