Search::Tools::Token - a token object returned from a TokenList
use Search::Tools::Tokenizer; my $tokenizer = Search::Tools::Tokenizer->new(); my $tokens = $tokenizer->tokenize('quick brown red dog'); while ( my $token = $tokens->next ) { # token isa Search::Tools::Token print "token = $token\n"; printf("str: %s, len = %d, u8len = %d, pos = %d, is_match = %d, is_hot = %d\n", $token->str, $token->len, $token->u8len, $token->pos, $token->is_match, $token->is_hot ); }
A Token represents one or more characters culled from a string by a Tokenizer.
Most of Search::Tools::Token is written in C/XS so if you view the source of this class you will not see much code. Look at the source for Tools.xs and search-tools.c if you are interested in the internals, or look at Search::Tools::TokenPP.
The characters in the token. Stringifies to the str() value with overloading.
The byte length of str().
The character length of str(). For ASCII, len() == u8len(). For non-ASCII UTF-8, u8len() < len().
The zero-based position in the original string.
Did the token match the re() in the Tokenizer.
Did the token match the heat_seeker in the Tokenizer.
Returns true value if the Token starts with an UPPER case UTF8 character or other common sentence-starting character.
Returns true value if the Token matches common sentence-ending punctuation.
Returns true value if the Token looks like a common English abbreviation.
Prints the internal XS attributes to stderr.
Set the is_hot() value.
Set the is_match() value.
Peter Karman <karman@cpan.org>
<karman@cpan.org>
Please report any bugs or feature requests to bug-search-tools at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=Search-Tools. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.
bug-search-tools at rt.cpan.org
You can find documentation for this module with the perldoc command.
perldoc Search::Tools
You can also look for information at:
RT: CPAN's request tracker
http://rt.cpan.org/NoAuth/Bugs.html?Dist=Search-Tools
AnnoCPAN: Annotated CPAN documentation
http://annocpan.org/dist/Search-Tools
CPAN Ratings
http://cpanratings.perl.org/d/Search-Tools
Search CPAN
http://search.cpan.org/dist/Search-Tools/
Copyright 2009 by Peter Karman.
This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install Search::Tools, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Search::Tools
CPAN shell
perl -MCPAN -e shell install Search::Tools
For more information on module installation, please visit the detailed CPAN module installation guide.