Search::Tools::TokenPP - a token object returned from a TokenList
use Search::Tools::Tokenizer; my $tokenizer = Search::Tools::Tokenizer->new(); my $tokens = $tokenizer->tokenize_pp('quick brown red dog'); while ( my $token = $tokens->next ) { # token isa Search::Tools::TokenPP print "token = $token\n"; printf("str: %s, len = %d, u8len = %d, pos = %d, is_match = %d, is_hot = %d\n", $token->str, $token->len, $token->u8len, $token->pos, $token->is_match, $token->is_hot ); }
A TokenPP represents one or more characters culled from a string by a Tokenizer.
TokenPP is a pure-Perl version of Token. See the Token docs for more details.
This class inherits from Search::Tools::Object. Only new or overridden methods are documented here.
The characters in the token. Stringifies to the str() value with overloading.
The byte length of str().
The character length of str(). For ASCII, len() == u8len(). For non-ASCII UTF-8, u8len() < len().
The zero-based position in the original string.
Did the token match the re() in the Tokenizer.
Did the token match the heat_seeker in the Tokenizer.
Returns true value if the Token matches common sentence-ending punctuation.
Set the is_hot() value.
Set the is_match() value.
Peter Karman <karman@cpan.org>
<karman@cpan.org>
Please report any bugs or feature requests to bug-search-tools at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Search-Tools. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.
bug-search-tools at rt.cpan.org
You can find documentation for this module with the perldoc command.
perldoc Search::Tools
You can also look for information at:
RT: CPAN's request tracker
http://rt.cpan.org/NoAuth/Bugs.html?Dist=Search-Tools
AnnoCPAN: Annotated CPAN documentation
http://annocpan.org/dist/Search-Tools
CPAN Ratings
http://cpanratings.perl.org/d/Search-Tools
Search CPAN
http://search.cpan.org/dist/Search-Tools/
Copyright 2009 by Peter Karman.
This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install Search::Tools, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Search::Tools
CPAN shell
perl -MCPAN -e shell install Search::Tools
For more information on module installation, please visit the detailed CPAN module installation guide.