Paulo Custodio


Parse::FSM::Lexer - Companion Lexer for the Parse::FSM parser


  use Parse::FSM::Lexer;
  $lex = Parse::FSM::Lexer->new;
  $lex = Parse::FSM::Lexer->new(@files);
  $lex->add_path(@dirs); @dirs = $lex->path;
  $full_path = $lex->path_search($file);

  $lex->from_list(sub {});
  # in a nearby piece of code
  use MyParser; # isa Parse::FSM::Driver;
  my $parser = MyParser->new;
  $parser->input(sub {$lex->get_token});
  eval {$parser->parse}; $@ and $lex->error($@);


This module implements a generic tokenizer that can be used by Parse::FSM parsers, and can also be used standalone independently of the parser.

It supports recursive file includes and takes track of current file name and line number. It keeps the path of search directories to search for input files.

The get_token method can be called by the input method of the parser to retrieves the next input token to parse.

The module can be used directly if the supplied tokenizer is enough for the application, but usually a derived class has to be written implementing a custom version of the tokenizer method.



Creates a new object. If an argument list is given, calls from_file for each of the file starting from the last, so that the files are read in the given order.



Returns the list of directories to search in sequence for source files.


Adds the given directories to the path searched for include files.

Searches for the given file name in the path created by add_path, returns the first full path name where the file can be found.

Returns the given input file name unchanged if:

  • the file is found in the current directory; or

  • the file is not found in any of the path directories.



Saves the current input context, searches for the given input file name in the path, opens the file and sets-up the object to read each line in sequence. At the end of the file input resumes to the place where it was when from_file was called.

Dies if the input file cannot be read, or if a file is included recursively, to avoid an infinite include loop.


Saves the current input context and sets-up the object to read each element of the passed input list. Each element either a text string or a code reference of an iterator that returns text strings. The iterator returns undef at the end of input.



Retrieves the next token from the input as an array reference containing token type and token value.

Returns undef on end of input.


Method responsible to match the next token from the given input string.

This method can be overridden by a child class in order to implement a different set of tokens to be retrieved from the input.

It is implemented with features from the Perl 5.010 regex engine:

  • one big regex with /\G.../gc to match from where the last match ended; the string to match is passed as a scalar reference, so that the position of last match pos() is preserved;

  • one sequence of (?:...|...) alternations for each token to be matched;

  • using (?>...) for each token to make sure there is no backtracking;

  • using capturing parentheses and embedded code evaluation (?{ [TYPE => $^N] }) to return the token value from the regex match;

  • using $^R as the value of the matched token;

    As the regex engine is not reentrant, any operation that may call another regex match (e.g. recursive file include) cannot be done inside the (?{ ... }) code block, and is done after the regex match by checking the $^R for special tokens.

  • using undef as the return of $^R to ignore a token, e.g. white space.

The default tokenizer recognizes and returns the following token types:

[STR => $value]

Perl-like single or double quoted string, $value contains the string without the quotes and with any backslash escapes resolved.

The string cannot span multiple input lines.

[NUM => $value]

Perl-like integer in decimal, hexadecimal, octal or binary notation, $value contains decimal value of the integer.

[NAME => $name]

Perl-like identifier name, i.e. word starting with a letter or underscore and followed by letters, underscores or digits.

[$token => $token]

All other characters except white space are returned in the form [$token=>$token], where $token is a single character or one of the following composed tokens: << >> == != >= <=

white space

All white space is ignored, i.e. the tokenizer returns undef.

[INCLUDE => $file]

Returned when a #include statement is recognized, causes the lexer to recursively include the file at the current input stream location.

[INPUT_POS => $file, $line_nr, $line_inc]

Returned when a #line statement is recognized, causes the lexer to set the current input location to the given $file, $line_nr and $line_inc.

[ERROR => $message]

Causes the lexer to call error with the given error message, can be used when the input cannot be tokenized.



Returns the current input file, undef if reading from a list.


Returns the current input line number, starting at 1.


Increment of line number on each new-line found, usually 1.


Dies with the given error message, indicating the place in the input source file where the error occurred.


Warns with the given error message, indicating the place in the input source file where the warning occurred.


See Parse::FSM