Marpa::Lex -- Utility Methods for Lexing
These routines are used internally by MDL to implement lexing of regexes and of q- and qq-quoted strings. They are documented here to make them available for general use within Marpa.
q-
qq-
my ( $regex, $token_length ) = Marpa::Lex::lex_regex( \$input_string, $lexeme_start );
Takes two required arguments. $string must be a reference to a string that might contain a regex. The regex will be expected to start at the position pointed to by pos $$string.
$string
pos $$string
$lexeme_start must be the start earleme of the regex for lexing purposes. In many cases (such as the removal of leading whitespace), it's useful to discard prefixes. If a prefix was removed prior to the call to lex_regex, $lexeme_start should be the location where the prefix started. If no prefix was removed, $lexeme_start will be the same as pos ${$string}.
$lexeme_start
lex_regex
pos ${$string}
How lex_regex delimits a regex is described in the MDL document. lex_regex returns the null array if no regex was found. If a regex was found, lex_regex returns an array of two elements. The first element is a string containing the regex, its delimiters, any postfix modifiers it had, and its qr- "operator" if there was one. The second is the regex's length for lexing purposes, which will include the length of any discarded prefix.
qr-
my ( $string, $token_length ) = Marpa::Lex::lex_q_quote( \$input_string, $lexeme_start );
Takes two required arguments, a string reference and a start earleme. The string reference must be to a string that might contain a q- or qq-quoted string. The q- or qq-quoted string will be expected to start at the position pointed to by pos ${$string}.
$lexeme_start must contain the start earleme of the quoted string for lexing purposes. In many cases (such as the removal of leading whitespace), it's useful to discard prefixes. If a prefix was removed prior to the call to lex_regex, $lexeme_start should be the location where the prefix started. If no prefix was removed, $lexeme_start should be the same as pos $$string.
How lex_q_quote delimits a q- or qq-quoted string is described in the MDL document. lex_q_quote returns the null array if no string was found. If a string was found, lex_q_quote returns an array of two elements. The first element is a string containing the q- or qq-quoted string, including the q- or qq- "operator" and the delimiters. The second is the quoted string's length for lexing purposes, which will include the length of any discarded prefix.
lex_q_quote
See the support section in the main module.
Jeffrey Kegler
Copyright 2007 - 2009 Jeffrey Kegler
This program is free software; you can redistribute it and/or modify it under the same terms as Perl 5.10.0.
To install Marpa, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Marpa
CPAN shell
perl -MCPAN -e shell install Marpa
For more information on module installation, please visit the detailed CPAN module installation guide.