Search::Tools::TokenListUtils - mixin methods for TokenList and TokenListPP
my $tokens = $tokenizer->tokenize( $string ); if ( $tokens->str eq $string) { print "string is same, before and after tokenize()\n"; } else { warn "I'm filing a bug report against Search::Tools right away!\n"; } my ($start_pos, $end_pos) = $tokens->get_window( 5, 20 ); # $start_pos probably == 0 # $end_pos probably == 25 my $slice = $tokens->get_window_pos( 5, 20 ); for my $token (@$slice) { print "token = $token\n"; }
Search::Tools::TokenListUtils contains pure-Perl methods inhertited by both Search::Tools::TokenList and Search::Tools::TokenListPP.
Returns a serialized version of the TokenList. If you haven't altered the TokenList since you got it from tokenize(), then str() returns a scalar string identical to (but not the same) the string you passed to tokenize().
Both Search::Tools::TokenList and TokenListPP are overloaded to stringify to the str() value.
Returns array with two values: start and end positions for the array of length size on either side of pos. Like taking a slice of the TokenList.
Note that size is the number of tokens not matches. So if you're looking for the number of "words", think about size*2.
Note too that size is the number of tokens on one side of pos. So the entire window width (length of the returned slice) is size*2 +/-1. The window is guaranteed to be bounded by matches.
If as_sentence is true, the window is shifted to try and match the first token prior to pos that returns true for is_sentence_start().
Like get_window() but returns an array ref of a slice of the TokenList containing Tokens.
Peter Karman <karman@cpan.org>
<karman@cpan.org>
Please report any bugs or feature requests to bug-search-tools at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Search-Tools. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.
bug-search-tools at rt.cpan.org
You can find documentation for this module with the perldoc command.
perldoc Search::Tools
You can also look for information at:
RT: CPAN's request tracker
http://rt.cpan.org/NoAuth/Bugs.html?Dist=Search-Tools
AnnoCPAN: Annotated CPAN documentation
http://annocpan.org/dist/Search-Tools
CPAN Ratings
http://cpanratings.perl.org/d/Search-Tools
Search CPAN
http://search.cpan.org/dist/Search-Tools/
Copyright 2009 by Peter Karman.
This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install Search::Tools, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Search::Tools
CPAN shell
perl -MCPAN -e shell install Search::Tools
For more information on module installation, please visit the detailed CPAN module installation guide.