NAME
String::Tokenizer - A simple string tokenizer.
SYNOPSIS
use String::Tokenizer;
# create the tokenizer and tokenize input
my $tokenizer = String::Tokenizer->new("((5+5) * 10)", '+*()');
# create tokenizer
my $tokenizer = String::Tokenizer->new();
# ... then tokenize the string
$tokenizer->tokenize("((5 + 5) - 10)", '()');
# will print '(, (, 5, +, 5, ), -, 10, )'
print join ", " => $tokenizer->getTokens();
# get a token iterator
my $i = $tokenizer->iterator();
while ($i->hasNextToken()) {
my $next = $i->nextToken();
# peek ahead at the next token
my $look_ahead = $i->lookAheadToken();
# ...
# skip the next 2 tokens
$i->skipTokens(2);
# ...
# then backtrack 1 token
my $previous = $i->prevToken();
# ...
# get the current token
my $current = $i->currentToken();
# ...
}
DESCRIPTION
A simple string tokenizer which takes a string and splits it on whitespace. It also optionally takes a string of characters to use as delimiters, and returns them with the token set as well. This allows for splitting the string in many different ways.
This is a very basic tokenizer, so more complex needs should be either addressed with a custom written tokenizer or post-processing of the output generated by this module. Basically, this will not fill everyones needs, but it spans a gap between simple split / /, $string
and the other options that involve much larger and complex modules.
Also note that this is not a lexical analyser. Many people confuse tokenization with lexical analysis. A tokenizer mearly splits its input into specific chunks, a lexical analyzer classifies those chunks. Sometimes these two steps are combined, but not here.
METHODS
- new ($string, $delimiters)
-
If you do not supply any parameters, nothing happens, the instance is just created. But if you do supply parameters, they are passed on to the
tokenize
method and that method is run. For information about those arguments, seetokenize
below. - tokenize ($string, $delimiters)
-
Takes a
$string
to tokenize, and optionally a set of$delimiter
characters to facilitate the tokenization. The$string
parameter is obvious, the$delimiter
parameter is not as transparent.$delimiter
is a string of characters, these characters are then seperated into individual characters and are used to split the$string
with. So given this string:(5 + (100 * (20 - 35)) + 4)
The
tokenize
method without a$delimiter
parameter would return the following comma seperated list of tokens:'(5', '+', '(100', '*', '(20', '-', '35))', '+', '4)'
However, if you were to pass the following set of delimiters
(, )
totokenize
, you would get the following comma seperated list of tokens:'(', '5', '+', '(', '100', '*', '(', '20', '-', '35', ')', ')', '+', '4', ')'
We now can differentiate the parens from the numbers, and no globbing occurs. If you wanted to allow for optionally leaving out the whitespace in the expression, like this:
(5+(100*(20-35))+4)
as some languages do. Then you would give this delimiter
+*-()
to arrive at the same result. - getTokens
-
Simply returns the array of tokens. It returns an array-ref in scalar context.
- iterator
-
Returns a String::Tokenizer::Iterator instance, see below for more details.
INNER CLASS
A String::Tokenizer::Iterator instance is returned from the String::Tokenizer's iterator
method and serves as yet another means of iterating through an array of tokens. The simplest way would be to call getTokens
and just manipulate the array yourself, or push the array into another object. However, iterating through a set of tokens tends to get messy when done manually. So here I have provided the String::Tokenizer::Iterator to address those common token processing idioms. It is basically a bi-directional iterator which can look ahead, skip and be reset to the begining.
NOTE: String::Tokenizer::Iterator is an inner class, which means that only String::Tokenizer objects can create an instance of it. That said, if String::Tokenizer::Iterator's new
method is called from outside of the String::Tokenizer package, an exception is thrown.
- new ($tokens_array_ref)
-
This accepts an array reference of tokens and sets up the iterator. This method can only be called from within the String::Tokenizer package, otherwise an exception will be thrown.
- reset
-
This will reset the interal counter, bringing it back to the begining of the token list.
- hasNextToken
-
This will return true (1) if there are more tokens to be iterated over, and false (0) otherwise.
- hasPrevToken
-
This will return true (1) if the begining of the token list has been reached, and false (0) otherwise.
- nextToken
-
This dispenses the next available token, and move the internal counter ahead by one.
- prevToken
-
This dispenses the previous token, and moves the internal counter back by one.
- currentToken
-
This returns the current token, which will match the last token retrieved by
nextToken
. - lookAheadToken
-
This peeks ahead one token to the next one in the list. This item will match the next item dispensed with
nextToken
. This is a non-destructive look ahead, meaning it does not alter the position of the internal counter. - skipToken
-
This will jump the internal counter ahead by 1.
- skipTokens ($number_to_skip)
-
This will jump the internal counter ahead by
$number_to_skip
.
TO DO
The Java StringTokenizer class allows for a token to be tokenized further, therefore breaking it up more and including the results into the current token stream. I have never used this feature in this class, but I can see where it might be a useful one. This may be in the next release if it works out.
BUGS
None that I am aware of. Of course, if you find a bug, let me know, and I will be sure to fix it.
CODE COVERAGE
I use Devel::Cover to test the code coverage of my tests, below is the Devel::Cover report on this module's test suite.
------------------------------------- ------ ------ ------ ------ ------ ------ ------
File stmt branch cond sub pod time total
------------------------------------- ------ ------ ------ ------ ------ ------ ------
/String/Tokenizer.pm 100.0 100.0 60.0 100.0 100.0 10.1 95.6
t/10_String_Tokenizer_test.t 100.0 n/a n/a 100.0 n/a 5.9 100.0
t/20_String_Tokenizer_Iterator_test.t 100.0 100.0 n/a 100.0 n/a 84.0 100.0
------------------------------------- ------ ------ ------ ------ ------ ------ ------
Total 100.0 100.0 60.0 100.0 100.0 100.0 97.4
------------------------------------- ------ ------ ------ ------ ------ ------ ------
SEE ALSO
The interface and workings of this module are based largely on the StringTokenizer class from the Java standard library.
Below is a short list of other modules that might be considered similar to this one. If this module does not suit your needs, you might look at one of these.
- String::Tokeniser
-
Along with being a tokenizer, it also provides a means of moving through the resulting tokens, allowing for skipping of tokens and such. But this module looks as if it hasnt been updated from 0.01 and that was uploaded in since 2002.
- Parse::Tokens
-
This one hasn't been touched since 2001, although it did get up to version 0.27. It looks to lean over more towards the parser side than a basic tokenizer.
- Text::Tokenizer
-
This one looks more up to date (updated as recently as March 2004), but is both a lexical analyzer and a tokenizer. It also uses XS, mine is pure perl. This is something maybe to look into if you were to need a more beefy solution that what String::Tokenizer provides.
AUTHOR
stevan little, <stevan@iinteractive.com>
COPYRIGHT AND LICENSE
Copyright 2004 by Infinity Interactive, Inc.
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.