Parse::FSM::Lexer - Companion Lexer for the Parse::FSM parser
use Parse::FSM::Lexer; $lex = Parse::FSM::Lexer->new; $lex = Parse::FSM::Lexer->new(@files); $lex->add_path(@dirs); @dirs = $lex->path; $full_path = $lex->path_search($file); $lex->from_file($filename); $lex->from_list(@input); $lex->from_list(sub {}); $lex->get_token; $lex->error(@message); $lex->file; $lex->line_nr; # in a nearby piece of code use MyParser; # isa Parse::FSM::Driver; my $parser = MyParser->new; $parser->input(sub {$lex->get_token}); eval {$parser->parse}; $@ and $lex->error($@);
This module implements a generic tokenizer that can be used by Parse::FSM parsers, and can also be used stand alone independently of the parser.
It supports recursive file includes and takes track of current file name and line number. It keeps the path of search directories to search for input files.
The get_token method can be called by the input method of the parser to retrieves the next input token to parse.
get_token
input
The module can be used directly if the supplied tokenizer is enough for the application, but usually a derived class has to be written implementing a custom version of the tokenizer method.
tokenizer
Creates a new object. If an argument list is given, calls from_file for each of the file starting from the last, so that the files are read in the given order.
from_file
Returns the list of directories to search in sequence for source files.
Adds the given directories to the path searched for include files.
Searches for the given file name in the path created by add_path, returns the first full path name where the file can be found.
path
add_path
Returns the given input file name unchanged if:
the file is found in the current directory; or
the file is not found in any of the path directories.
Saves the current input context, searches for the given input file name in the path, opens the file and sets-up the object to read each line in sequence. At the end of the file input resumes to the place where it was when from_file was called.
Dies if the input file cannot be read, or if a file is included recursively, to avoid an infinite include loop.
Saves the current input context and sets-up the object to read each element of the passed input list. Each element either a text string or a code reference of an iterator that returns text strings. The iterator returns undef at the end of input.
undef
Retrieves the next token from the input as an array reference containing token type and token value.
Returns undef on end of input.
Method responsible to match the next token from the given input string.
This method can be overriden by a child class in order to implement a different set ot tokens to be retrieved from the input.
It is implemented with features from the Perl 5.010 regex engine:
one big regex with /\G.../gc to match from where the last match ended; the string to match is passed as a scalar reference, so that the position of last match pos() is preserved;
/\G.../gc
pos()
one sequence of (?:...|...) alternations for each token to be matched;
(?:...|...)
using (?>...) for each token to make sure there is no backtracking;
(?>...)
using capturing parentheses and embedded code evaluation (?{ [TYPE => $^N] }) to return the token value from the regex match;
(?{ [TYPE => $^N] })
using $^R as the value of the matched token;
$^R
As the regex engine is not reentrant, any operation that may call another regex match (e.g. recursive file include) cannot be done inside the (?{ ... }) code block, and is done after the regex match by checking the $^R for special tokens.
(?{ ... })
using undef as the return of $^R to ignore a token, e.g. white space.
The default tokenizer recognizes and returns the following token types:
Perl-like single or double quoted string, $value contains the string without the quotes and with any backslash escapes resolved.
$value
The string cannot span multiple input lines.
Perl-like integer in decimal, hexadecimal, octal or binary notation, $value contains decimal value of the integer.
Perl-like identifier name, i.e. word starting with a letter or underscore and followed by letters, underscores or digits.
All other characters except white space are returned in the form [$token=>$token], where $token is a single character or one of the following composed tokens: << >> == != >= <=
[$token=>$token]
$token
All white space is ignored, i.e. the tokenizer returns undef.
Returned when a #include statement is recognized, causes the lexer to recursively include the file at the current input stream location.
#include
Returned when a #line statement is recognized, causes the lexer to set the current input location to the given $file, $line_nr and $line_inc.
#line
$file
$line_nr
$line_inc
Causes the lexer to call error with the given error message, can be used when the input cannot be tokenized.
error
Returns the current input file, undef if reading from a list.
Returns the current input line number, starting at 1.
Increment of line number on each new-line found, usually 1.
Dies with the given error message, indicating the place in the input source file where the error occured.
Warns with the given error message, indicating the place in the input source file where the warning occured.
See Parse::FSM
To install Parse::FSM, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Parse::FSM
CPAN shell
perl -MCPAN -e shell install Parse::FSM
For more information on module installation, please visit the detailed CPAN module installation guide.