NAME

Text::Balanced::Marpa - Extract delimited text sequences from strings

Synopsis

        #!/usr/bin/env perl

        use strict;
        use warnings;

        use Text::Balanced::Marpa ':constants';

        # -----------

        my($count)  = 0;
        my($parser) = Text::Balanced::Marpa -> new
        (
                open    => ['<:' ,'[%'],
                close   => [':>', '%]'],
                options => nesting_is_fatal | print_warnings,
        );
        my(@text) =
        (
                q|<: a :>|,
                q|a [% b <: c :> d %] e|,
                q|a <: b <: c :> d :> e|, # nesting_is_fatal triggers an error here.
        );

        my($result);

        for my $text (@text)
        {
                $count++;

                print "Parsing |$text|\n";

                $result = $parser -> parse(text => \$text);

                print join("\n", @{$parser -> tree -> tree2string}), "\n";
                print "Parse result: $result (0 is success)\n";

                if ($count == 3)
                {
                        print "Deliberate error: Failed to parse |$text|\n";
                        print 'Error number: ', $parser -> error_number, '. Error message: ',
                                        $parser -> error_message, "\n";
                }

                print '-' x 50, "\n";
        }

See scripts/synopsis.pl.

This is the printout of synopsis.pl:

        Parsing |<: a :>|
        Parsed text:
        root. Attributes: {}
           |--- open. Attributes: {text => "<:"}
           |   |--- string. Attributes: {text => " a "}
           |--- close. Attributes: {text => ":>"}
        Parse result: 0 (0 is success)
        --------------------------------------------------
        Parsing |a [% b <: c :> d %] e|
        Parsed text:
        root. Attributes: {}
           |--- string. Attributes: {text => "a "}
           |--- open. Attributes: {text => "[%"}
           |   |--- string. Attributes: {text => " b "}
           |   |--- open. Attributes: {text => "<:"}
           |   |   |--- string. Attributes: {text => " c "}
           |   |--- close. Attributes: {text => ":>"}
           |   |--- string. Attributes: {text => " d "}
           |--- close. Attributes: {text => "%]"}
           |--- string. Attributes: {text => " e"}
        Parse result: 0 (0 is success)
        --------------------------------------------------
        Parsing |a <: b <: c :> d :> e|
        Error: Parse failed. Opened delimiter <: again before closing previous one
        Text parsed so far:
        root. Attributes: {}
           |--- string. Attributes: {text => "a "}
           |--- open. Attributes: {text => "<:"}
               |--- string. Attributes: {text => " b "}
        Parse result: 1 (0 is success)
        Deliberate error: Failed to parse |a <: b <: c :> d :> e|
        Error number: 2. Error message: Opened delimiter <: again before closing previous one
        --------------------------------------------------

See also scripts/tiny.pl and scripts/traverse.pl.

Description

Text::Balanced::Marpa provides a Marpa::R2-based parser for extracting delimited text sequences from strings. The text outside and inside the delimiters, and delimiters themselves, are all stored as nodes in a tree managed by Tree.

Nested strings, with the same or different delimiters, are stored as daughters of the nodes which hold the delimiters.

This module is a companion to Text::Delimited::Marpa. The differences are discussed in the "FAQ" below.

See the "FAQ" for various topics, including:

o UFT8 handling

See t/utf8.t.

o Escaping delimiters within the text

See t/escapes.t.

o Options to make nested and/or overlapped delimiters fatal errors

See t/colons.t.

o Using delimiters which are part of another delimiter

See t/escapes.t and t/perl.delimiters.

o Processing the tree-structured output

See scripts/traverse.pl.

o Emulating Text::Xslate's use of '<:' and ':>

See t/colons.t and t/percents.t.

o Implementing a really trivial HTML parser

See t/html.t.

In the same vein, see t/angle.brackets.t, for code where the delimiters are just '<' and '>'.

o Handling multiple sets of delimiters

See t/multiple.delimiters.t.

o Skipping (leading) characters in the input string

See t/skip.prefix.t.

o Implementing hard-to-read text strings as delimiters

See t/silly.delimiters.

Distributions

This module is available as a Unix-style distro (*.tgz).

See http://savage.net.au/Perl-modules/html/installing-a-module.html for help on unpacking and installing distros.

Installation

Install Text::Balanced::Marpa as you would any Perl module:

Run:

        cpanm Text::Balanced::Marpa

or run:

        sudo cpan Text::Balanced::Marpa

or unpack the distro, and then either:

        perl Build.PL
        ./Build
        ./Build test
        sudo ./Build install

or:

        perl Makefile.PL
        make (or dmake or nmake)
        make test
        make install

Constructor and Initialization

new() is called as my($parser) = Text::Balanced::Marpa -> new(k1 => v1, k2 => v2, ...).

It returns a new object of type Text::Balanced::Marpa.

Key-value pairs accepted in the parameter list (see corresponding methods for details [e.g. "text([$stringref])"]):

o close => $arrayref

An arrayref of strings, each one a closing delimiter.

The # of elements must match the # of elements in the 'open' arrayref.

See the "FAQ" for details and warnings.

A value for this option is mandatory.

Default: None.

o length => $integer

The maxiumum length of the input string to process.

This parameter works in conjunction with the pos parameter.

length can also be used as a key in the hash passed to "parse([%hash])".

See the "FAQ" for details.

Default: Calls Perl's length() function on the input string.

o next_few_limit => $integer

This controls how many characters are printed when displaying 'the next few chars'.

It only affects debug output.

Default: 20.

o open => $arrayref

An arrayref of strings, each one an opening delimiter.

The # of elements must match the # of elements in the 'open' arrayref.

See the "FAQ" for details and warnings.

A value for this option is mandatory.

Default: None.

o options => $bit_string

This allows you to turn on various options.

options can also be used as a key in the hash passed to "parse([%hash])".

Default: 0 (nothing is fatal).

See the "FAQ" for details.

o pos => $integer

The offset within the input string at which to start processing.

This parameter works in conjunction with the length parameter.

pos can also be used as a key in the hash passed to "parse([%hash])".

See the "FAQ" for details.

Note: The first character in the input string is at pos == 0.

Default: 0.

o text => $stringref

This is a reference to the string to be parsed. A stringref is used to avoid copying what could potentially be a very long string.

text can also be used as a key in the hash passed to "parse([%hash])".

Default: \''.

Methods

bnf()

Returns a string containing the grammar constructed based on user input.

close()

Get the arrayref of closing delimiters.

See also "open()".

See the "FAQ" for details and warnings.

'close' is a parameter to "new()". See "Constructor and Initialization" for details.

delimiter_action()

Returns a hashref, where the keys are delimiters and the values are either 'open' or 'close'.

delimiter_frequency()

Returns a hashref where the keys are opening and closing delimiters, and the values are the # of times each delimiter appears in the input stream.

The value is incremented for each opening delimiter and decremented for each closing delimiter.

error_message()

Returns the last error or warning message set.

Error messages always start with 'Error: '. Messages never end with "\n".

Parsing error strings is not a good idea, ever though this module's format for them is fixed.

See "error_number()".

error_number()

Returns the last error or warning number set.

Warnings have values < 0, and errors have values > 0.

If the value is > 0, the message has the prefix 'Error: ', and if the value is < 0, it has the prefix 'Warning: '. If this is not the case, it's a reportable bug.

Possible values for error_number() and error_message():

o 0 => ""

This is the default value.

o 1/-1 => "Last open delimiter: $lexeme_1. Unexpected closing delimiter: $lexeme_2"

If "error_number()" returns 1, it's an error, and if it returns -1 it's a warning.

You can set the option overlap_is_fatal to make it fatal.

o 2/-2 => "Opened delimiter $lexeme again before closing previous one"

If "error_number()" returns 2, it's an error, and if it returns -2 it's a warning.

You can set the option nesting_is_fatal to make it fatal.

o 3/-3 => "Ambiguous parse. Status: $status. Terminals expected: a, b, ..."

This message is only produced when the parse is ambiguous.

If "error_number()" returns 3, it's an error, and if it returns -3 it's a warning.

You can set the option ambiguity_is_fatal to make it fatal.

o 4 => "Backslash is forbidden as a delimiter character"

This preempts some types of sabotage.

This message can never be just a warning message.

o 5 => "Single-quotes are forbidden in multi-character delimiters"

This limitation is due to the syntax of Marpa's DSL.

This message can never be just a warning message.

o 6/-6 => "Parse exhausted"

If "error_number()" returns 6, it's an error, and if it returns -6 it's a warning.

You can set the option exhaustion_is_fatal to make it fatal.

o 7 => 'Single-quote is forbidden as an escape character'

This limitation is due to the syntax of Marpa's DSL.

This message can never be just a warning message.

o 8 => "There must be at least 1 pair of open/close delimiters"

This message can never be just a warning message.

o 9 => "The # of open delimiters must match the # of close delimiters"

This message can never be just a warning message.

o 10 => "Unexpected event name 'xyz'"

Marpa has trigged an event and it's name is not in the hash of event names derived from the BNF.

This message can never be just a warning message.

o 11 => "The code does not handle these events simultaneously: a, b, ..."

The code is written to handle single events at a time, or in rare cases, 2 events at the same time. But here, multiple events have been triggered and the code cannot handle the given combination.

This message can never be just a warning message.

See "error_message()".

escape_char()

Get the escape char.

known_events()

Returns a hashref where the keys are event names and the values are 1.

length([$integer])

Here, the [] indicate an optional parameter.

Get or set the length of the input string to process.

See also the "FAQ" and "pos([$integer])".

'length' is a parameter to "new()". See "Constructor and Initialization" for details.

matching_delimiter()

Returns a hashref where the keys are opening delimiters and the values are the corresponding closing delimiters.

new()

See "Constructor and Initialization" for details on the parameters accepted by "new()".

next_few_chars($stringref, $offset)

Returns a substring of $s, starting at $offset, for use in debug messages.

See next_few_limit([$integer]).

next_few_limit([$integer])

Here, the [] indicate an optional parameter.

Get or set the number of characters called 'the next few chars', which are printed during debugging.

'next_few_limit' is a parameter to "new()". See "Constructor and Initialization" for details.

open()

Get the arrayref of opening delimiters.

See also "close()".

See the "FAQ" for details and warnings.

'open' is a parameter to "new()". See "Constructor and Initialization" for details.

options([$bit_string])

Here, the [] indicate an optional parameter.

Get or set the option flags.

For typical usage, see scripts/synopsis.pl.

See the "FAQ" for details.

'options' is a parameter to "new()". See "Constructor and Initialization" for details.

parse([%hash])

Here, the [] indicate an optional parameter.

This is the only method the user needs to call. All data can be supplied when calling "new()".

You can of course call other methods (e.g. "text([$stringref])" ) after calling "new()" but before calling parse().

The optional hash takes these ($key => $value) pairs (exactly the same as for "new()"):

o length => $integer
o options => $bit_string
o pos => $integer
o text => $stringref

Note: If a value is passed to parse(), it takes precedence over any value with the same key passed to "new()", and over any value previously passed to the method whose name is $key. Further, the value passed to parse() is always passed to the corresponding method (i.e. whose name is $key), meaning any subsequent call to that method returns the value passed to parse().

See scripts/samples.pl.

Returns 0 for success and 1 for failure.

If the value is 1, you should call "error_number()" to find out what happened.

pos([$integer])

Here, the [] indicate an optional parameter.

Get or set the offset within the input string at which to start processing.

See also the "FAQ" and "length([$integer])".

'pos' is a parameter to "new()". See "Constructor and Initialization" for details.

text([$stringref])

Here, the [] indicate an optional parameter.

Get or set a reference to the string to be parsed.

'text' is a parameter to "new()". See "Constructor and Initialization" for details.

tree()

Returns an object of type Tree, which holds the parsed data.

Obviously, it only makes sense to call tree() after calling parse().

See scripts/traverse.pl for sample code which processes this tree's nodes.

FAQ

What are the differences between Text::Balanced::Marpa and Text::Delimited::Marpa?

I think this is shown most clearly by getting the 2 modules to process the same string. So, using this as input:

        'a <:b <:c:> d:> e <:f <: g <:h:> i:> j:> k'

Output from Text::Balanced::Marpa's scripts/tiny.pl:

        (#   2) |          1         2         3         4         5         6         7         8         9
                |0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
        Parsing |Skip me ->a <:b <:c:> d:> e <:f <: g <:h:> i:> j:> k|. pos: 10. length: 42
        Parse result: 0 (0 is success)
        root. Attributes: {text => "", uid => "0"}
            |--- text. Attributes: {text => "a ", uid => "1"}
            |--- open. Attributes: {text => "<:", uid => "2"}
            |    |--- text. Attributes: {text => "b ", uid => "3"}
            |    |--- open. Attributes: {text => "<:", uid => "4"}
            |    |    |--- text. Attributes: {text => "c", uid => "5"}
            |    |--- close. Attributes: {text => ":>", uid => "6"}
            |    |--- text. Attributes: {text => " d", uid => "7"}
            |--- close. Attributes: {text => ":>", uid => "8"}
            |--- text. Attributes: {text => " e ", uid => "9"}
            |--- open. Attributes: {text => "<:", uid => "10"}
            |    |--- text. Attributes: {text => "f ", uid => "11"}
            |    |--- open. Attributes: {text => "<:", uid => "12"}
            |    |    |--- text. Attributes: {text => " g ", uid => "13"}
            |    |    |--- open. Attributes: {text => "<:", uid => "14"}
            |    |    |    |--- text. Attributes: {text => "h", uid => "15"}
            |    |    |--- close. Attributes: {text => ":>", uid => "16"}
            |    |    |--- text. Attributes: {text => " i", uid => "17"}
            |    |--- close. Attributes: {text => ":>", uid => "18"}
            |    |--- text. Attributes: {text => " j", uid => "19"}
            |--- close. Attributes: {text => ":>", uid => "20"}
            |--- text. Attributes: {text => " k", uid => "21"}

Output from Text::Delimited::Marpa's scripts/tiny.pl:

        (#   2) |          1         2         3         4         5         6         7         8         9
                |0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
        Parsing |Skip me ->a <:b <:c:> d:> e <:f <: g <:h:> i:> j:> k|. pos: 10. length: 42
        Parse result: 0 (0 is success)
        root. Attributes: {end => "0", length => "0", start => "0", text => "", uid => "0"}
            |--- span. Attributes: {end => "22", length => "9", start => "14", text => "b <:c:> d", uid => "1"}
            |    |--- span. Attributes: {end => "18", length => "1", start => "18", text => "c", uid => "2"}
            |--- span. Attributes: {end => "47", length => "18", start => "30", text => "f <: g <:h:> i:> j", uid => "3"}
                 |--- span. Attributes: {end => "43", length => "10", start => "34", text => " g <:h:> i", uid => "4"}
                      |--- span. Attributes: {end => "39", length => "1", start => "39", text => "h", uid => "5"}

Another example, using the same input string, but manually processing the tree nodes. Parent-daughter relationships are here represented by indentation.

Output from Text::Balanced::Marpa's scripts/traverse.pl:

                |          1         2         3         4         5
                |012345678901234567890123456789012345678901234567890
        Parsing |a <:b <:c:> d:> e <:f <: g <:h:> i:> j:> k|.
        Span  Text
           1  |a |
           2  |<:|
           3    |b |
           4    |<:|
           5      |c|
           6    |:>|
           7    | d|
           8  |:>|
           9  | e |
          10  |<:|
          11    |f |
          12    |<:|
          13      | g |
          14      |<:|
          15        |h|
          16      |:>|
          17      | i|
          18    |:>|
          19    | j|
          20  |:>|
          21  | k|

Output from Text::Delimited::Marpa's scripts/traverse.pl:

                |          1         2         3         4         5
                |012345678901234567890123456789012345678901234567890
        Parsing |a <:b <:c:> d:> e <:f <: g <:h:> i:> j:> k|.
        Span  Start  End  Length  Text
           1      4   12       9  |b <:c:> d|
           2      8    8       1    |c|
           3     20   37      18  |f <: g <:h:> i:> j|
           4     24   33      10    | g <:h:> i|
           5     29   29       1      |h|

Where are the error messages and numbers described?

See "error_message()" and "error_number()".

How do I escape delimiters?

By backslash-escaping the first character of all open and close delimiters which appear in the text.

As an example, if the delimiters are '<:' and ':>', this means you have to escape all the '<' chars and all the colons in the text.

The backslash is preserved in the output.

If you don't want to use backslash for escaping, or can't, you can pass a different escape character to "new()".

See t/escapes.t.

How do the length and pos parameters to new() work?

The recognizer - an object of type Marpa::R2::Scanless::R - is called in a loop, like this:

        for
        (
                $pos = $self -> recce -> read($stringref, $pos, $length);
                $pos < $length;
                $pos = $self -> recce -> resume($pos)
        )

"pos([$integer])" and "length([$integer])" can be used to initialize $pos and $length.

Note: The first character in the input string is at pos == 0.

See https://metacpan.org/pod/distribution/Marpa-R2/pod/Scanless/R.pod#read for details.

Does this package support Unicode/UTF8?

Yes. See t/escapes.t, t/multiple.quotes.t and t/utf8.t.

Does this package handler Perl delimiters (e.g. q|..|, qq|..|, qr/../, qw/../)?

See t/perl.delimiters.t.

Warning: Calling mutators after calling new()

The only mutator which works after calling new() is "text([$stringref])".

In particular, you can't call "escape_char()", "open()" or "close()" after calling "new()". This is because parameters passed to new() are interpolated into the grammar before parsing begins. And that's why the docs for those methods all say 'Get the...' and not 'Get and set the...'.

To make the code work, you would have to manually call _validate_open_close(). But even then a lot of things would have to be re-initialized to give the code any hope of working.

What is the format of the 'open' and 'close' parameters to new()?

Each of these parameters takes an arrayref as a value.

The # of elements in the 2 arrayrefs must be the same.

The 1st element in the 'open' arrayref is the 1st user-chosen opening delimiter, and the 1st element in the 'close' arrayref must be the corresponding closing delimiter.

It is possible to use a delimiter which is part of another delimiter.

See scripts/samples.pl. It uses both '<' and '<:' as opening delimiters and their corresponding closing delimiters are '>' and ':>'. Neat, huh?

What are the possible values for the 'options' parameter to new()?

Firstly, to make these constants available, you must say:

        use Text::Balanced::Marpa ':constants';

Secondly, more detail on errors and warnings can be found at "error_number()".

Thirdly, for usage of these option flags, see t/angle.brackets.t, t/colons.t, t/escapes.t, t/multiple.quotes.t, t/percents.t and scripts/samples.pl.

Now the flags themselves:

o nothing_is_fatal

This is the default.

nothing_is_fatal has the value of 0.

o print_errors

Print errors if this flag is set.

print_errors has the value of 1.

o print_warnings

Print various warnings if this flag is set:

o The ambiguity status and terminals expected, if the parse is ambiguous
o See "error_number()" for other warnings which might be printed

Ambiguity is not, in and of itself, an error. But see the ambiguity_is_fatal option, below.

It's tempting to call this option warnings, but Perl already has use warnings, so I didn't.

print_warnings has the value of 2.

o print_debugs

Print extra stuff if this flag is set.

print_debugs has the value of 4.

o overlap_is_fatal

This means overlapping delimiters cause a fatal error.

So, setting overlap_is_fatal means '{Bold [Italic}]' would be a fatal error.

I use this example since it gives me the opportunity to warn you, this will not do what you want if you try to use the delimiters of '<' and '>' for HTML. That is, '<i><b>Bold Italic</i></b>' is not an error because what overlap are '<b>' and '</i>' BUT THEY ARE NOT TAGS. The tags are '<' and '>', ok? See also t/html.t.

overlap_is_fatal has the value of 8.

o nesting_is_fatal

This means nesting of identical opening delimiters is fatal.

So, using nesting_is_fatal means 'a <: b <: c :> d :> e' would be a fatal error.

nesting_is_fatal has the value of 16.

o ambiguity_is_fatal

This makes "error_number()" return 3 rather than -3.

ambiguity_is_fatal has the value of 32.

o exhaustion_is_fatal

This makes "error_number()" return 6 rather than -6.

exhaustion_is_fatal has the value of 64.

How do I print the tree built by the parser?

See "Synopsis".

How do I make use of the tree built by the parser?

See scripts/traverse.pl. It is a copy of t/html.t with tree-walking code instead of test code.

How is the parsed data held in RAM?

The parsed output is held in a tree managed by Tree.

The tree always has a root node, which has nothing to do with the input data. So, even an empty imput string will produce a tree with 1 node. This root has an empty hashref associated with it.

Nodes have a name and a hashref of attributes.

The name indicates the type of node. Names are one of these literals:

o close
o open
o root
o text

For 'open' and 'close', the delimiter is given by the value of the 'text' key in the hashref.

The (key => value) pairs in the hashref are:

o text => $string

If the node name is 'open' or 'close', $string is the delimiter.

If the node name is 'text', $string is the verbatim text from the document.

Verbatim means, for example, that backslashes in the input are preserved.

Try:

        perl -Ilib scripts/samples.pl info

How is HTML/XML handled?

The tree does not preserve the nested nature of HTML/XML.

Post-processing (valid) HTML could easily generate another view of the data.

But anyway, to get perfect HTML you'd be grabbing the output of Marpa::R2::HTML, right?

See scripts/traverse.pl and t/html.t for a trivial HTML parser.

What is the homepage of Marpa?

http://savage.net.au/Marpa.html.

That page has a long list of links.

How do I run author tests?

This runs both standard and author tests:

        shell> perl Build.PL; ./Build; ./Build authortest

TODO

o Advanced error reporting

See https://jeffreykegler.github.io/Ocean-of-Awareness-blog/individual/2014/11/delimiter.html.

Perhaps this could be a sub-class?

o I8N support for error messages
o An explicit test program for parse exhaustion

See Also

Text::Delimited::Marpa.

Tree and Tree::Persist.

Text::Balanced.

MarpaX::Demo::SampleScripts - for various usages of Marpa::R2, but not of this module.

Machine-Readable Change Log

The file Changes was converted into Changelog.ini by Module::Metadata::Changes.

Version Numbers

Version numbers < 1.00 represent development versions. From 1.00 up, they are production versions.

Thanks

Thanks to Jeffrey Kegler, who wrote Marpa and Marpa::R2.

And thanks to rns (Ruslan Shvedov) for writing the grammar for double-quoted strings used in MarpaX::Demo::SampleScripts's scripts/quoted.strings.02.pl. I adapted it to HTML (see scripts/quoted.strings.05.pl in that module), and then incorporated the grammar into GraphViz2::Marpa, and - after more extensions - into this module.

Lastly, thanks to Robert Rothenberg for Const::Exporter, a module which works the same way Perl does.

Repository

https://github.com/ronsavage/Text-Balanced-Marpa

Support

Email the author, or log a bug on RT:

https://rt.cpan.org/Public/Dist/Display.html?Name=Text::Balanced::Marpa.

Author

Text::Balanced::Marpa was written by Ron Savage <ron@savage.net.au> in 2014.

Marpa's homepage: http://savage.net.au/Marpa.html.

My homepage: http://savage.net.au/.

Copyright

Australian copyright (c) 2014, Ron Savage.

        All Programs of mine are 'OSI Certified Open Source Software';
        you can redistribute them and/or modify them under the terms of
        The Artistic License 2.0, a copy of which is available at:
        http://opensource.org/licenses/alphabetical.