The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
KinoSearch-0.20_01
River stage two • 9 direct dependents • 10 total dependents
1 ++

Changes for version 0.20_01 - 2007-02-26

  • KinoSearch 0.20 is a major rewrite, adding many new features. It also breaks backwards compatibility in a number of ways.
  • Two key features, UTF-8 support and custom sorting, were not possible to implement while preserving backwards compatibility. Once the decision was made to proceed with them, breaking all existing installations, it made little sense to proceed by half measures, so the API has been given a significant overhaul.
  • KinoSearch has always carried an "alpha code" warning; it is being invoked for this release. While it will continue to carry the "alpha" warning for a short while longer, the point of jamming so many changes into one release is to cause disruption only once; once the code in 0.20 proves itself, hopefully no more backwards incompatible changes will be needed any time soon.
  • New behaviors:
    • KinoSearch now uses UTF-8 for all input and output, throughout the entire library. This affects many classes, but particularly those under Analysis, Highlight, and QueryParser.
    • The default scoring algorithm has changed subtly -- aggressive per-field boosting is no longer important or even desirable. The old behavior is available from KinoSearch::Contrib::LongFieldSim.
  • New public classes:
    • KinoSearch::Schema
    • KinoSearch::Schema::Field
    • KinoSearch::InvIndex
    • KinoSearch::Analysis::Token
    • KinoSearch::Search::RangeFilter
    • KinoSearch::Search::SortSpec
    • KinoSearch::Search::Similarity
    • KinoSearch::Contrib::LongFieldSim
  • New documentation:
    • KinoSearch::Docs::NFS
  • Removed classes:
    • KinoSearch::Document::Doc
    • KinoSearch::Document::Field
    • KinoSearch::Search::Hit
  • Renamed classes:
    • KinoSearch::Store::InvIndex => KinoSearch::Store::Folder
    • KinoSearch::Store::FSInvIndex => KinoSearch::Store::FSFolder
    • KinoSearch::Store::RAMInvIndex => KinoSearch::Store::RAMFolder
  • Updated documentation:
    • KinoSearch
    • KinoSearch::Docs::DevGuide
    • KinoSearch::Docs::FileFormat
    • KinoSearch::Docs::Tutorial
  • Classes with API changes:
    • KinoSearch::InvIndexer o new() - Args changed.
      • create - Removed.
      • analyzer - Removed.
      • lock_id - Added.
      • o spec_field() - Removed. o new_doc() - Removed. o add_doc() - Args changed.
        • Takes a hashref rather than a Doc object.
        • Accepts optional labeled param 'boost'.
      • o delete_docs_by_term() - Removed. o delete_by_term() - Added. (Behavior differs subtly from delete_docs_by_term()).
    • KinoSearch::Searcher o new() - args changed.
      • analyzer - Removed.
      • o search() - Now calls Hits->seek before returning Hits object. Args changed.
        • offset - Added.
        • num_wanted - Added.
        • sort_spec - Added.
    • KinoSearch::Search::Hits o Now comes pre-seeked, courtesy of changes to Searcher. o seek() - No longer triggers new number crunching if requested values can be accomodated using results of prior search. o fetch_hit() - Removed. o create_excerpts() - Now puts multiple excerpts under $hit->{excerpts} rather than one under $hit->{excerpt}.
    • KinoSearch::Search::MultiSearcher o new() - Args changed.
      • schema - Added.
      • analyzer - Removed.
    • KinoSearch::Highlight::Highlighter o new() - Args changed.
      • fields - Added.
      • excerpt_length - Now specified in characters rather than bytes.
      • excerpt_field - Removed.
      • pre_tag - Removed.
      • post_tag - Removed.
    • KinoSearch::QueryParser::QueryParser o new() - Args changed.
      • schema - Added.
      • default_field - Removed.
      • analyzer - No longer required -- now used to override schema.
    • KinoSearch::Analysis::TokenBatch o new() - Args changed.
      • text - Added.
      • o next() - Returns a Token instead of a boolean. o reset() - Added. o add_many_tokens() - Added. o set_text(), get_text(), set_start_offset(), get_start_offset(), set_end_offset(), get_end_offset(), set_pos_inc(), get_pos_inc - All removed.
  • Internal changes:
    • Large-scale refactoring has taken place. The most significant changes are...
    • OO framework imposed on C code via boilerplater.pl, with KinoSearch::Util::Obj as the base class.
    • Charmonizer added.
    • perlapi functions and data structures replaced whenever possible.
    • Lots of classes, especially under KinoSearch::Index, reorganized around Schema and SegInfo.
    • Many tests added, removed, or revised to accomodate changes in the main library code.
    • C code moved to dedicated files.
    • Build.PL custom code moved to buildlib/KinoSearchBuild.pm
  • File Format:
    • Significantly redesigned. The most visible change is that the segments file is now encoded using YAML rather than an arbitrary binary format.
    • Old indexes cannot be read and must be regenerated.
  • Locking
    • write.lock files now located in the index directory rather than under /tmp.
    • Commit locks are no longer needed due to file format changes.
    • Stale write locks are now removed without warning.

Documentation

boilerplater.pl
Generate boilerplate OO code for KinoSearch
dump_index
dump the contents of an index
KinoSearch::Docs::DevGuide
Hacking/debugging KinoSearch.
KinoSearch::Docs::FileFormat
Overview of invindex file format.
KinoSearch::Docs::NFS
Managing invindexes on NFS.
KinoSearch::Docs::Tutorial
Sample indexing and search applications.

Modules

KinoSearchBuild
Module::Build subclass for KinoSearch
KinoSearch
Search engine library.
KinoSearch::Analysis::Analyzer
Base class for analyzers.
KinoSearch::Analysis::LCNormalizer
Convert input to lower case.
KinoSearch::Analysis::PolyAnalyzer
Multiple analyzers in series.
KinoSearch::Analysis::Stemmer
Reduce related words to a shared root.
KinoSearch::Analysis::Stopalizer
Suppress a "stoplist" of common words.
KinoSearch::Analysis::Token
Unit of text.
KinoSearch::Analysis::TokenBatch
A collection of tokens.
KinoSearch::Analysis::Tokenizer
Customizable tokenizing.
KinoSearch::Contrib::LongFieldSim
Similarity optimized for long fields.
KinoSearch::Highlight::Encoder
Encode excerpted text.
KinoSearch::Highlight::Formatter
Format highlighted bits within excerpts.
KinoSearch::Highlight::Highlighter
Create and highlight excerpts.
KinoSearch::Highlight::SimpleHTMLEncoder
Encode a few HTML entities.
KinoSearch::Highlight::SimpleHTMLFormatter
Surround highlight bits with tags.
KinoSearch::Index::Term
String of text associated with a field.
KinoSearch::InvIndex
An inverted index.
KinoSearch::InvIndexer
Build inverted indexes.
KinoSearch::QueryParser::QueryParser
Transform a string into a Query object.
KinoSearch::Schema
User-created specification for an inverted index.
KinoSearch::Schema::FieldSpec
Define a field's behavior.
KinoSearch::Search::BooleanQuery
Match boolean combinations of Queries.
KinoSearch::Search::Hits
Access search results.
KinoSearch::Search::MultiSearcher
Aggregate results from multiple searchers.
KinoSearch::Search::PhraseQuery
Match ordered list of Terms.
KinoSearch::Search::Query
Base class for search queries.
KinoSearch::Search::QueryFilter
Build a filter based on results of a query.
KinoSearch::Search::RangeFilter
Filter search results by range of values.
KinoSearch::Searcher::SearchClient
Connect to a remote SearchServer.
KinoSearch::Search::SearchServer
Make a Searcher remotely accessible.
KinoSearch::Search::Similarity
Calculate how closely two things match.
KinoSearch::Search::SortSpec
Specify a custom sort order for search results.
KinoSearch::Search::TermQuery
Match individual Terms.
KinoSearch::Searcher
Execute searches.
KinoSearch::Store::FSFolder
File System implementation of Folder.
KinoSearch::Store::Folder
Abstract class representing a directory.
KinoSearch::Store::RAMFolder
In-memory Folder.
KinoSearch::Util::Class
Class-building utility.
KinoSearch::Util::ToolSet
Namespace pollution.

Provides

KinoSearch::Index::CompoundFileReader
in lib/KinoSearch/Index/CompoundFileReader.pm
KinoSearch::Index::CompoundFileWriter
in lib/KinoSearch/Index/CompoundFileWriter.pm
KinoSearch::Index::DelDocs
in lib/KinoSearch/Index/DelDocs.pm
KinoSearch::Index::DocReader
in lib/KinoSearch/Index/DocReader.pm
KinoSearch::Index::DocVector
in lib/KinoSearch/Index/DocVector.pm
KinoSearch::Index::DocWriter
in lib/KinoSearch/Index/DocWriter.pm
KinoSearch::Index::IndexFileNames
in lib/KinoSearch/Index/IndexFileNames.pm
KinoSearch::Index::IndexReader
in lib/KinoSearch/Index/IndexReader.pm
KinoSearch::Index::MultiReader
in lib/KinoSearch/Index/MultiReader.pm
KinoSearch::Index::MultiTermDocs
in lib/KinoSearch/Index/MultiTermDocs.pm
KinoSearch::Index::MultiTermList
in lib/KinoSearch/Index/MultiTermList.pm
KinoSearch::Index::PostingsWriter
in lib/KinoSearch/Index/PostingsWriter.pm
KinoSearch::Index::SegInfo
in lib/KinoSearch/Index/SegInfo.pm
KinoSearch::Index::SegInfos
in lib/KinoSearch/Index/SegInfos.pm
KinoSearch::Index::SegReader
in lib/KinoSearch/Index/SegReader.pm
KinoSearch::Index::SegTermDocs
in lib/KinoSearch/Index/SegTermDocs.pm
KinoSearch::Index::SegTermList
in lib/KinoSearch/Index/SegTermList.pm
KinoSearch::Index::SegWriter
in lib/KinoSearch/Index/SegWriter.pm
KinoSearch::Index::TermDocs
in lib/KinoSearch/Index/TermDocs.pm
KinoSearch::Index::TermInfo
in lib/KinoSearch/Index/TermInfo.pm
KinoSearch::Index::TermList
in lib/KinoSearch/Index/TermList.pm
KinoSearch::Index::TermListCache
in lib/KinoSearch/Index/TermListCache.pm
KinoSearch::Index::TermListReader
in lib/KinoSearch/Index/TermListReader.pm
KinoSearch::Index::TermListWriter
in lib/KinoSearch/Index/TermListWriter.pm
KinoSearch::Index::TermVector
in lib/KinoSearch/Index/TermVector.pm
KinoSearch::Index::TermVectorsReader
in lib/KinoSearch/Index/TermVectorsReader.pm
KinoSearch::Index::TermVectorsWriter
in lib/KinoSearch/Index/TermVectorsWriter.pm
KinoSearch::Search::BooleanClause
in lib/KinoSearch/Search/BooleanClause.pm
KinoSearch::Search::BooleanScorer
in lib/KinoSearch/Search/BooleanScorer.pm
KinoSearch::Search::BooleanWeight
in lib/KinoSearch/Search/BooleanQuery.pm
KinoSearch::Search::CacheDFSource
in lib/KinoSearch/Search/MultiSearcher.pm
KinoSearch::Search::FieldDoc
in lib/KinoSearch/Search/FieldDoc.pm
KinoSearch::Search::FieldDocCollator
in lib/KinoSearch/Search/FieldDocCollator.pm
KinoSearch::Search::HitCollector
in lib/KinoSearch/Search/HitCollector.pm
KinoSearch::Search::HitQueue
in lib/KinoSearch/Search/HitQueue.pm
KinoSearch::Search::PhraseScorer
in lib/KinoSearch/Search/PhraseScorer.pm
KinoSearch::Search::PhraseWeight
in lib/KinoSearch/Search/PhraseQuery.pm
KinoSearch::Search::ScoreDoc
in lib/KinoSearch/Search/ScoreDoc.pm
KinoSearch::Search::Scorer
in lib/KinoSearch/Search/Scorer.pm
KinoSearch::Search::Searchable
in lib/KinoSearch/Search/Searchable.pm
KinoSearch::Search::SortCollector
in lib/KinoSearch/Search/SortCollector.pm
KinoSearch::Search::SortedHitQueue
in lib/KinoSearch/Search/SortedHitQueue.pm
KinoSearch::Search::TermScorer
in lib/KinoSearch/Search/TermScorer.pm
KinoSearch::Search::TermWeight
in lib/KinoSearch/Search/TermQuery.pm
KinoSearch::Search::TopDocCollector
in lib/KinoSearch/Search/TopDocCollector.pm
KinoSearch::Search::TopDocs
in lib/KinoSearch/Search/TopDocs.pm
KinoSearch::Search::Weight
in lib/KinoSearch/Search/Weight.pm
KinoSearch::Store::FSFileDes
in lib/KinoSearch/Store/FSFileDes.pm
KinoSearch::Store::FileDes
in lib/KinoSearch/Store/FileDes.pm
KinoSearch::Store::InStream
in lib/KinoSearch/Store/InStream.pm
KinoSearch::Store::Lock
in lib/KinoSearch/Store/Lock.pm
KinoSearch::Store::OutStream
in lib/KinoSearch/Store/OutStream.pm
KinoSearch::Store::RAMFileDes
in lib/KinoSearch/Store/RAMFileDes.pm
KinoSearch::Util::BitVector
in lib/KinoSearch/Util/BitVector.pm
KinoSearch::Util::ByteBuf
in lib/KinoSearch/Util/ByteBuf.pm
KinoSearch::Util::CClass
in lib/KinoSearch/Util/CClass.pm
KinoSearch::Util::DynVirtualTable
in lib/KinoSearch/Util/DynVirtualTable.pm
KinoSearch::Util::Hash
in lib/KinoSearch/Util/Hash.pm
KinoSearch::Util::IntMap
in lib/KinoSearch/Util/IntMap.pm
KinoSearch::Util::Obj
in lib/KinoSearch/Util/Obj.pm
KinoSearch::Util::PriorityQueue
in lib/KinoSearch/Util/PriorityQueue.pm
KinoSearch::Util::SortExternal
in lib/KinoSearch/Util/SortExternal.pm
KinoSearch::Util::StringHelper
in lib/KinoSearch/Util/StringHelper.pm
KinoSearch::Util::ToStringUtils
in lib/KinoSearch/Util/ToStringUtils.pm
KinoSearch::Util::VArray
in lib/KinoSearch/Util/VArray.pm
KinoSearch::Util::VerifyArgs
in lib/KinoSearch/Util/VerifyArgs.pm
KinoSearch::Util::ViewByteBuf
in lib/KinoSearch/Util/ViewByteBuf.pm
KinoSearch::Util::YAML
in lib/KinoSearch/Util/YAML.pm

Examples

Other files