The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
Search-Tools-1.007
River stage two • 17 direct dependents • 34 total dependents
7 ++
/ Search::Tools::TokenListUtils

NAME

Search::Tools::TokenListUtils - mixin methods for TokenList and TokenListPP

SYNOPSIS

 my $tokens = $tokenizer->tokenize( $string );
 if ( $tokens->str eq $string) {
    print "string is same, before and after tokenize()\n";
 }
 else {
    warn "I'm filing a bug report against Search::Tools right away!\n";
 }
 
 my ($start_pos, $end_pos) = $tokens->get_window( 5, 20 );
 # $start_pos probably == 0
 # $end_pos probably   == 25
 
 my $slice = $tokens->get_window_pos( 5, 20 );
 for my $token (@$slice) {
    print "token = $token\n";
 }

DESCRIPTION

Search::Tools::TokenListUtils contains pure-Perl methods inhertited by both Search::Tools::TokenList and Search::Tools::TokenListPP.

METHODS

str

Returns a serialized version of the TokenList. If you haven't altered the TokenList since you got it from tokenize(), then str() returns a scalar string identical to (but not the same as) the string you passed to tokenize().

Both Search::Tools::TokenList and TokenListPP are overloaded to stringify to the str() value.

get_window( pos [, size, as_sentence] )

Returns array with two values: start and end positions for the array of length size on either side of pos. Like taking a slice of the TokenList.

Note that size is the number of tokens not matches. So if you're looking for the number of "words", think about size*2.

Note too that size is the number of tokens on one side of pos. So the entire window width (length of the returned slice) is size*2 +/-1. The window is guaranteed to be bounded by matches.

If as_sentence is true, the window is shifted to try and match the first token prior to pos that returns true for is_sentence_start().

get_window_tokens( pos [, size] )

Like get_window() but returns an array ref of a slice of the TokenList containing Tokens.

as_sentences([stringified])

Returns a reference to an array of arrays, where each child array is a "sentence" worth of Token objects. You can stringify each sentence array like:

 my $sentences = $tokenlist->as_sentences;
 for my $s (@$sentences) {
     printf("sentence: %s\n", join("", map {"$_"} @$s));
 }

If you pass a single true value to as_sentences(), then the array returned will consist of plain scalar strings with whitespace normalized.

AUTHOR

Peter Karman <karman@cpan.org>

BUGS

Please report any bugs or feature requests to bug-search-tools at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Search-Tools. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

SUPPORT

You can find documentation for this module with the perldoc command.

    perldoc Search::Tools

You can also look for information at:

COPYRIGHT

Copyright 2009 by Peter Karman.

This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself.