Search::Tools::Snipper - extract terms in context


 use Search::Tools;
 my $query = qw/ quick dog /;
 my $text  = 'the quick brown fox jumped over the lazy dog';

 my $s = Search::Tools->snipper(
     occur       => 3,
     context     => 8,
     word_len    => 5,
     max_chars   => 300,
     query       => $query

 print $s->snip( $text );


Search::Tools::Snipper extracts terms and their context from a larger block of text. The larger block may be plain text or HTML/XML.


new( query => query )

Instantiate a new object. query must be either a scalar string or a Search::Tools::Query object

Many of the following methods are also available as key/value pairs to new().


Called internally by new().

as_sentences Experimental feature

Attempt to extract a snippet that starts at a sentence boundary.


The number of snippets that should be returned by snip().

Available via new().


The number of context words to include in the snippet.

Available via new().


The maximum number of characters (not bytes! under Perl >= 5.8) to return in a snippet. NOTE: This is only used to test whether test is worth snipping at all, or if no terms are found.

See also show() and ignore_length().

Available via new().


The estimated average word length used in combination with context(). You can usually ignore this value.

Available via new().


Boolean flag indicating whether snip() should succeed no matter what, or if it should give up if no snippets were found. Default is 1 (true).

If no matches are found, the first max_chars of the snippet are returned.

Available via new().


Boolean flag indicating whether snip() should escape any HTML/XML markup in the resulting snippet or not. Default is 0 (false).

Available via new().


Boolean flag indicating whether snip() should attempt to remove any HTML/XML markup in the original text before snipping is applied. Default is 0 (false).

Available via new().


The CODE ref used by the snip() method for actually extracting snippets. You can use your own snipper function if you want (though if you have a better snipper algorithm than the ones in this module, why not share it?). If you go this route, have a look at the source code for snip() to see how snipper() is used.

Available via new().


There are different algorithms used internally for snipping text. They are, in order of speed:


Just grabs the first max_chars characters and returns it, doing a little clean up to prevent partial words from ending the snippet and (optionally) escaping the text.


Fastest for single-word queries.


Most accurate, for both single-word and phrase queries, although it relies on a HeatMap in order to locate phrases.

See also the use_pp feature.

offset (default)

Same as re but optimized slightly to look at a substr of text.


The regular expression algorithm. Will match phrases exactly.


The name of the internal snipper function used. In case you're curious.


Boolean flag indicating whether the snipper() value should always be used, regardless of the type of query keyword. Default is 0 (false).

Available via new().


The number of snips made by the Snipper object.


Boolean flag indicating whether multiple whitespace characters should be collapsed into a single space. A whitespace character is defined as anything that Perl's \s pattern matches, plus the nobreak space (\xa0). Default is 1 (true).

Available via new().

use_pp( n )

Set to a true value to use Tokenizer->tokenize_pp() and TokenListPP and TokenPP instead of the XS versions of the same. XS is the default and is much faster, but harder to modify or subclass.

Available via new().


Boolean flag. If set to false (default) then max_chars is respected. If set to true, max_chars is ignored.

Available via new().


Boolean flag. If set to true (default), individual terms within a phrase are considered a match. If false, only match if individual terms have a proximity distance of 1.

snip( text )

Return a snippet of text from text that matches query plus context() words of context. Matches are case insensitive.

The snippet returned will be in UTF-8 encoding, regardless of the encoding of text.


Peter Karman <karman at cpan dot org>


Based on the HTML::HiLiter regular expression building code, originally by the same author, copyright 2004 by Cray Inc.

Thanks to Atomic Learning for sponsoring the development of this module.


Copyright 2006 by Peter Karman.

This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself.