NAME

Text::Fab - A powerful, general-purpose document expansion framework (currently Control Flow, Loops, and Caching are unimplemented {???})

SYNOPSIS

Three input files, written for the default (dbg0) callbacks:

## -- File: config.fab --
## Declare which configuration keys are lists
#append Fab/list_keys chapters 1
#append Fab/list_keys joiners/comma 1
## Set some configuration variables
#set site_title=My Awesome Book
#append chapters chapter1 chapter2 chapter3
#append joiners/comma ,
## Define the inheritance graph of the namespaces
#set_parents ChapterLayout BaseLayout
## The final book inherits from _main's structure and from ChapterLayout
#set_parents Book _main ChapterLayout

## -- File: components.fab --
## Reusable components, defined in namespaces
#target_section title in ChapterLayout
<h1>
#include configkey joiners/comma site_title
</h1>
#end_section
#target_section table_of_contents in ChapterLayout
<p>Chapters:
#include configkey joiners/comma chapters
</p>
#end_section
#target_section footer in BaseLayout
<footer>Copyright 2025</footer>
#end_section

## -- File: main.fab --
## The main assembly script.  The default output target is _main:body
#emb title
#emb table_of_contents
#emb footer

Processing them, then assembling the book:

use Text::Fab;
my $fab = Text::Fab->new;
$fab->include__text(filename => $_) for qw(config.fab components.fab main.fab);
print $fab->assemble('Book:body'), "\n";

prints

<h1>
My Awesome Book</h1>
<p>Chapters:
chapter1,chapter2,chapter3</p>
<footer>Copyright 2025</footer>

Book:body is not defined, so it is found in _main, the first parent of Book. The promises inside it are resolved late, starting from Book, the namespace the body was requested in: title and table_of_contents are found in ChapterLayout, and footer in BaseLayout (the parent of ChapterLayout). #include configkey runs while the file is read, so the configuration it uses must be set before (here: in config.fab).

Not yet implemented {???}: the original version of this example also used the constructs below. The default callbacks do not support them (yet); they are kept here as a reminder of the intended syntax.

#append Fab/list_keys chapters css_files joiners/lines  # several names at once, trailing comment
#append joiners/lines "\n"                              # a quoted value with an escape
<h1>#emb _main:title</h1>                               # a directive inside a line: needs a custom interleaver
#target_section all_chapters in ChapterLayout
    #= FOREACH chapter IN C('chapters') ...             # needs a Chunk Preprocessor that understands "#="
        #emb @chapter:body
    #= END #
#end_section

DESCRIPTION

Text::Fab is not a templating engine; it is a framework for building your own. At its core, it is more or less a minimal "dumb" state machine that exposes a stable API of primitive operations. Most of the "smart" logic for parsing syntax, processing text, and creating complex control structures is delegated to user-configurable components. (However, a certain minimal level of smartness is a — switchable off — default to handle the most common cases.) As a result, it knows nothing about the processed “language”'s syntax or semantics.

The main design goal is that all state of Fab is explicitly introspectable: there is no hidden magic. The entire state of the system is contained in the Configuration hash and the collection of already constructed Output Sections. What you see is what you get.

The Fab’s purpose is to factor out the common needs of all the configurable-on-the-fly document processing engines; more precisely, it focuses on the needs orthogonal to the particular syntax of a particular problem domain. (This architecture aims to overcome the experiences with designing “simple” configurable systems — which turn out to be not scalable, so all the approaches to make them operable crumble under their own weight.)

In short: this attempts to abstract out all the complexity of scalable configurability. The resulting “tools” support architecture with a top→down approach to recognizing the “reconfiguration directives”.

————————————————————————————————————

By default, Text::Fab reads input and filters it through a specified preprocessing engine (which defaults to pass-through-unchanged) to the standard output. Its power comes from executing the interleaved-in-the-input directives that are tools for controlling the processing engine, reshuffling content, and templating.

The process is straightforward: Text::Fab asks a De-Interleaver callback "Where is the next control directive?". It then (optionally, see below) passes the text chunk before that location to a Chunk Preprocessor (the "stomach"). Finally, it allows the Parser to execute the directive (typically, the Parser would convert the details of the directive into calls to zero or more of Text::Fab's primitive API methods to alter its internal state). This cycle repeats until the input is exhausted.

Furthermore, as an extra configurable step, the De-Interleaver may need to report that a particular directive “is a comment-to-ignore”. In this case, Fab would postpone processing the text chunk before that location, instead merging it with the following chunk(s) before passing to the Pre-Processor. So the process’ cycle is: the De-Interleaver finds the next directive. The Comment Recognizer may then optionally identify a comment, allowing for the fusion of the preceding and succeeding text chunks. Depending on this: now — or later — this final chunk is then passed to the Chunk Preprocessor. Finally, the Parser executes the directive found above, which may alter the system's state for the next cycle.

Finally, this “structured” (see below) pre-processed data created this way is “merged” into the final output text stream following flexible “hierarchical” rules.

Preview of architecture

To explain what this engine is supposed to simplify, consider a (somewhat contrived) example of document preparation. You have a prepared source of a document; it may consists of several files which should be joined by certain rules; each file is a mix of (recognizable) chunks of markup, LaTeX and HTML; etc. You want to produce a best-effort PDF file, several LaTeX files prepared for publication in various outlets, a readable plain-text version, and a web page.

To add insult to injury, many parts of the document should be displayed differently in different parts of the output. (For example, a section header may be quadriplicated: as a TOC entry, a running page header, as a PDF’s TOC entry, and its rendition in the main text.) And the details of prepocessing some embedded promise (say, MathChunk with the “content” x²) may need to be “massaged” into (the same) string x² for the PDF header, or to $x^2$ when handled inside TOC (or running headers), or to $$x^2$$ in the header rendered “as a part of main text” (well, I warned you that the example is contrived!).

On top of this, the rendition may depend on a mix of recipes inherited from different sources: the default solution “from the vendor”, the enhancements designed by your organization, enhancements found on the web, the requirements of the journal, the legal requirements of the country, your personal enhancements to these etc.

How to handle processing of all this mess through one pass over the initial document? The architecture of Text::Fab encodes the intermediate steps of processing into 6 components, One of them — The Configuration — is very special since all other components (except the content of output sections) are in fact encoded inside it. The aim of these components is to provide carpets under which one can conveniently sweep all the parts of complexity-dust which “require coding” — leaving only “the elementary core dependencies” exposed to both the user and the developer. The intent is that after this sweeping, all (or: better be all?) the necessary configuration/fine-tuning might be done not by writing/editing new code, but by issuing processing directives.

Configuration: A multi-level hash with all the control data needed to process the flow, as well as the temporary variables.
Undo engine: The (structured) list of undo actions to execute at the end of certain processing steps (possibly including undoing changes to the configuration).
Input chain: The inclusion-chain of input files currently in processing (but not yet finished).
Output sections: They are named, and encode chunks of semi-digested input. Each consists of plain text strings and of “promises” to be resolved later, all in one (1-dimensional) list. Each promise contains a name of recipe and the name of section to process (and possibly hints on the “name resolution” of recipes).
Namespaces: They form a DAG encoding “which namespace overrides what” (with the root overriding them all, etc.). Each one resolves certain recipe names to the code to implement these recipes. (Note that if A overrides B, this may be interpreted as “B contains the default actions for A”. — Following the metaphor of inheritance, we call B “a parent” for A.)
A chain of “lenses”: Each lens encodes “the aim of the currently active pipeline” of processing the promises in the sections. Each request to set a new aim is translated (see "Massaging of lens names") to just a particular vertex in the DAG of namespaces, a “virtual root” — so the name resolution is restricted to it and “its defaults”: whatever namespaces it overrides. The resulting vertex is appended the chain of lenses.

Example of usage

With our still contrived example, suppose that the initial pass over the document converted to an output section §base with 4 promises (and nothing else):

#emb secname §sec1name
#emb sectext §sec1text
#emb secname §sec2name
#emb sectext §sec2text

(with §sec1name etc. being the names of other sections), and 4 “referenced above” sections — as well as some other sections which may be referenced from those 4 — etc. (These 4 sections are minimally preprocessed — just enough to recognize what region in the input files goes where (but one can as well also run the preprocessing steps common for all output formats). When processing to LaTeX: one of the first calls would establish the outermost lens to be LaTeX. When one needs to produce TOC, one processes §base as

#emb[aim=as_TOC_lens] as_TOC §base

The installed “recipes for lenses” would (presumably) change the requested root as_TOC_lens to LaTeX/as_TOC_lens. The resulting stack of lenses is now LaTeX LaTeX/as_TOC_lens. And the active lens LaTeX/as_TOC_lens is (presumably) set up to inherit from as_TOC_lens — which sets the name resolution to process sectext promises to completely ignore the corresponding output sections (which are now read for input!), while processing sections of secname promises into their “maximally compact output format”. And on top of this, LaTeX/as_TOC_lens would also arrange to wrap the result into the necessary LaTeX macros.

Should not we list how exactly it is done ???  A promise C<MathChunk> inside §sec2name would be (presumably) resolved to a
C<LaTeX/as_TOC_lens/MathChunk> recipe.  This recipe would pass the content (HOW???  How can it delegate to
C<as_TOC_lens/MathChunk>?  By another C<aim=>?) into a promise C<compact/MathChunk>, and wrap the result into a suitable
LaTeX macro.

One processes to PDF_TOC likewise — however, here one needs to transform from the content of §sec2name not to LaTeX macros, but to Unicode plain text. — This may be arranged as above, through a certain installed sub-preprocessor used as a pipe in “an output section” → “an output section”.

When one processes “the main body” of the output, one calls

#emb[aim=as_BODY_lens] as_BODY §base

which would lead (likewise) to the output sections of both secnames and sectexts to be fully processed (with suitable macros wrapping section names — which are also processed into a compact form to be put into the running headers).

Note that how the fact “that the output is to LaTeX” is set up by one top-level call (inside a certain 1-liner output section — WHICH???)! If one wants conversion to PDF or HTML instead and the necessary translators of ”elementary subblocks” are already present and configured, all that one needs to do is to replace this 1-liner by another one. Due to a change to the toplevel lens, the names of the following lenses on the stack would change accordingly (although the specified “aims” when installing these lenses did not change!) — which would change the namespace resolution “as needed”, so the same promises have a chance to depend completely differently. In particular, they may insert “the same promises as for LaTeX” (but which may now depend differently), or they may replace/enhance them by absolutely different promises specialized for HTML processing.

When one need to reformat the LaTeX output for a particular journal, one may override how the names of promises are converted to their “implementation” by including the needed implementations/enhancements into a new namespace(s), inserting them accordingly into the DAG of namespaces.

Massaging of lens names

The translation aim → lens above defaults to id (then the names of lenses conincide with “the request”). More generally, it depends on the current chain of lenses — controlled on the registered list of “which lenses affect which”. If a lens Foo affects Bar, then the result is the lens Foo/Bar; additionally, in the longer chain A, B, Foo, C, D, Bar, if A affects Foo/Bar, then the result is¹⁾ A/(Foo/Bar).

¹⁾ Caveat usor: in the chain C<A, Foo, Foo/Baz> a request for C<Bar> may result in²⁾ C<(Foo/Baz)/Bar>.
   (May later propagate to C<((Foo/Baz)/Bar)/Gee>.)  Since it is only the user/developer who could declare such “affections”,
   it is their responsibility to insert
²⁾ May we ever need to clean C<Foo/(Foo/Baz)/Bar>???

TODO: DAG affecting affections — need inheritance on Foo above???

TODO: If a new aim PDF3.0 appears, and one declared that it affects PDFbookmarks — which has been affecting TeXmath: if the inheritance of PDF3.0/PDFbookmarks on PDFbookmarks is declared, then it is fine: the request to TeXmath inside such a chain would still create PDFbookmarks/TeXmath — which (hopefully) would be processing things correctly. (And if not, then it is the responsibility of whoever implemented this “new aim” to recode how TeXmath behaves in presence of PDF3.0/PDFbookmarks, and declare the corresponding “affects” declaration).

On the othe hand, if the inheritance of PDF3.0/PDFbookmarks on PDFbookmarks is not declared (this may happen for example when the recipes in this namespace are ), then to enable the transparent handling (as above) one should use the other mechanism for “affecting to PDFbookmarks/TeXmath” (now with TeXmath “being a joker”): declare ???

THE PROCESSING MODEL

First of all, the inclusion mechanism handles a stack of currently opened-for-processing files; these files are “combined as usual” into one input stream. The active file chain is reflected in the configuration key opened_includes. In addition to this, the Fab uses three more data collections.

The configuration directives may affect how the preprocessor handles the chunks of its input, whether “the comments interleaved into the stream” split these chunks etc., and may modify how the “directives interleaved into the input stream” are recognized (and even what these directives mean). The grouping directives control which changes to configuration are undone and when. The output directives may insert into the output section "an order to inline another section" (a promise, which is not executed until much later). Another kind of output directive allows switching the target section for the output of the preprocessor.

(It makes sense to also focus on specific parts of the Configuration Hash: the Control Stack which manages conditional handling and looping [Planned], Grouping Stack which handles undoing changes to configuration, and Caching Engine which helps to avoid the penalties of constantly recalculating data dependent only on the configuration stack [Planned].)

After all the sections have been constructed, they may be “joined arbitrarily” into the final document. This assembly process is controlled by the rules how to choose a named section if “multiple flavors” have been defined. This allows a flexible hierarchical system of “overridable templates”. Such templates together with the rules of “namespace resolution” build the logic of processing that ultimately generates the final document(s).

The Configuration Hash

The heart of Text::Fab is a single Configuration hash. It is a live data structure that directives can modify during processing. It controls everything, from user-defined variables to the very components (parsers, etc.) that define the language's behavior. By default, all internal configuration used by the framework itself is stored under the Fab/ top-level key. (Later we are going to discuss how to modify this default, and how this prefix changes during the input processing. However, in examples below we assume that this hasn't been changed.)

Configuration values can be scalars, lists, or hashes. Special hashes, Fab/list_keys and Fab/hash_keys, are used to declare which keys should be treated as lists or hashes by the primitive operations. new() registers Fab/hash_keys, Fab/list_keys and Fab/lens_affects as hash keys.

Fab's own settings are read through the stack of prefixes Fab/call_stack_prefixes (see "Configuration Prefix and Call Stack API"): for a setting such as list_keys, Text::Fab looks at PREFIX/list_keys for each prefix on the stack, the most recent prefix first. This applies to list_keys, hash_keys and lens_affects (here a name is looked up in each prefix's hash in turn, so a recursive filter can add or override single entries), and to hash_key_sort_order, callbacks, callbacks_bundle/*, inheritance_graph, group_flavors, group_nesting, blinders/*, max_recursion_depth, output_handle and output_postpone. NS__set_parents writes under the most recent prefix. Only state kept by Text::Fab itself has fixed names: the stack of prefixes, Fab/call_stack_prefixes and Fab/call_stack_filter_name, and the stack of output targets, Fab/output_target_stack.

The Output Sections

A section is a named buffer that holds already preprocessed content. A section's name is a pair of strings: a namespace and a basename (e.g., Chapter1:body). The default namespace is _main.

It makes sense to imagine that a section consists of two interleaved lists, each of its specific type of content:

Plain text chunks, which have been processed by the Chunk Preprocessor.
embed placeholders, which are promises to inline another section during the final Assembly Phase. This other section is described by its basename as well as some extra data (these data are used in the calculation of its namespace, which is going to be performed later).

Target Section Stacking and Scoping

#target_section <basename [in <namespace>]>: Switches the active output target to the specified section, pushing it onto the stack of targets (see "Output" below). If namespace is omitted, defaults to the current target's namespace (_main if there is none). Autovivifies the section buffer if it does not already exist. The basename and the namespace are arbitrary non-space tokens, so lens namespaces such as LaTeX/as_TOC or (LaTeX/as_TOC)/Math can be used; anything else on the line halts with E_SYNTAX_ERROR.
#end_section: Pops the current target from the stack, restoring the output stream to the enclosing target. On an empty stack, and on the base entry of postponed output, it does nothing.
#create_section <basename [in <namespace>]>: Initializes or clears the specified section buffer without redirecting current preprocessor output.

Output

The output targets are kept as a stack in the Configuration hash, Fab/output_target_stack: a list of [$namespace, $basename], whose last entry is the current target. Preprocessed text and promises (#emb) are appended to the current target section. Since the stack is part of the configuration, #end_group restores it like any other key: a group with a full snapshot restores it, and a flavor with a list of keys restores it if the list names Fab/output_target_stack.

When the stack is empty, there is no target section: text goes directly to the output, the filehandle Fab/output_handle (by default \*STDOUT; it must be a glob reference, such as \*STDOUT, a lexical handle from open, or an IO::Handle object, since FreezeThaw cannot snapshot a bare *FH{IO}). A promise, however, can only be resolved during assembly, after all the input has been read; so it cannot go to direct output. The setting Fab/output_postpone decides what happens when such a promise is going to be written:

at_first_promise (the default): Text goes directly to the output until the first promise meets the empty stack; then _main:body is pushed as the base entry of postponed output, and the promise and all later output go there. So, with the default callbacks, include__text() of a file without directives works like cat.
at_start: Output is postponed from the start: when output is produced with an empty stack, the target _main:body is pushed as the base entry of postponed output, and everything goes there. (This was the only behaviour in 0.08 and earlier.)
error: Text goes directly to the output; a promise meeting the empty stack halts with E_PROMISE_IN_DIRECT_OUTPUT. (Promises inside a target section are fine.)

#end_section (out__end_section()) never pops the base entry of postponed output. The postponed output is printed to Fab/output_handle by out__flush(), which a driver calls when all the input has been read. (Both settings may be changed in the input, e.g. #set Fab/output_postpone=error, as long as this happens before the first output.)

Grouping: Scoped/Undoable Configuration Changes

(Below, we assume a particular #-based format for directives. The actual operation is agnostic of this format.)

#start_group [flavor_name]

Begins a scope. A certain set of keys in the Configuration hash are snapshotted. The flavor_name is used to look up the list of Configuration keys to memorize/restore under Fab/group_flavors/$flavor_name/keys (an array reference). If this key list is not defined for the flavor, the default behavior is to snapshot the entire Configuration hash. With a key list, only these keys are restored by #end_group; all other changes survive the group. The key list can be set up with directives, too (here only x is restored):

#append Fab/list_keys Fab/group_flavors/g/keys 1
#append Fab/group_flavors/g/keys x
#start_group g
#set x=2
#set y=2
#end_group g
Validates nesting constraints against the enclosing group flavor via
C<Fab/group_nesting/$flavor/allowed_parents>, halting with C<E_INVALID_GROUP_NESTING> if violated.
The active group flavor chain is mirrored in C<opened_groups>.

#end_group [flavor_name]

If a flavor_name is provided, it must match the flavor of the currently open group; otherwise, processing halts with E_MISMATCHED_END_GROUP. All changes made to the observed configuration keys are undone. If the optional name is not given, closes the most recently opened scope. Calling #end_group with no open scope halts with E_DANGLING_END_GROUP. Executes all postponed undo actions and restores the configuration snapshot.

Note: The nesting behavior of groups (e.g., whether flavor A must be nested within flavor B) is itself controlled by the Configuration hash.

DEFAULT (`dbg0`) DIRECTIVES REFERENCE

The default parser (_default_directive_parser) recognizes the following directives interleaved into the input stream.

Syntax. With the default callbacks, a directive is a line that starts with #: the interleaver recognizes # only as the first character of a line, so a # elsewhere (as in <h1>#emb title</h1>) is ordinary text. The directive line has the form #name or #name[options], followed by arguments separated by whitespace. There is no quoting, and a # among the arguments does not start a comment: it is passed on as an argument. A line that starts with ## is a comment. A line that starts with # but does not have this form (for example # text or #= ...) is reported to the error handler as malformed_directive. (Other syntaxes need other callbacks; their signatures are listed under PLACEHOLDERS below {???}.)

#set <key=<value>>

Sets a scalar configuration variable.

#append <key <values...>>

Appends values to a list key, appends string values to a scalar key, or merges an even-sized list of key-value pairs into a hash key.

#prepend <key <values...>>

Prepends values to a list key (preserving their order at the front of the list) or prepends string values to a scalar key. Prohibited on hash keys.

#pop <key [<count_or_subkeys...>]>

Pops elements from a list key by count (defaulting to 1), or deletes specified subkeys from a hash key. The count must be a non-negative integer (0 pops nothing); anything else halts with E_TYPE_MISMATCH.

#target_section <basename [in <namespace>]>

Redirects output to the specified section, pushing it onto the target stack.

#end_section

Pops the target stack and restores the enclosing output target (see "Output").

#create_section <basename [in <namespace>]>

Clears or creates a section buffer without redirecting output.

#emb[<options] <recipe> [<arg_sections...>]>

Inserts an unresolved embed placeholder/promise into the current section. Options in [...] accept comma-separated key-value pairs (e.g. aim=..., in=..., with_blinder=..., callbacks=...). (callbacks=... [Planned] {???}: the bundle is switched during the expansion, but assembly calls no callbacks yet, so it has no visible effect.) Bare #emb recipe acts as a direct section transclusion.

The recipe may be qualified (NS:recipe, or in=NS). An unqualified recipe is resolved late, during assembly: from the lens vertex if the promise sets aim=, otherwise starting from the namespace in which the enclosing section was requested (which may be a child of the namespace where the section's text was found).

Operands after recipe are bound to $ARGV0, $ARGV1, etc. in the recipe body during assembly. They are bound when the promise is expanded, in the context of the caller: NS:name starts its resolution at NS (lenses are ignored); an unqualified name starts at the caller's namespace; $ARGVn passes on the caller's own n-th operand. As a stopgap, when the promise switches the lens (aim=...), every operand must be tagged: old=name resolves as seen from the caller, new=name from the new lens vertex; an untagged operand on such a promise halts with E_UNTAGGED_OPERAND. (This may change once the semantic of operands is settled.)

#emb[aim=as_TOC] toc_entry old=sec1 new=sec7

#set_parents <namespace <parents...>>

Declares inheritance edges in the DAG. Validates cycle freedom (halting with E_CYCLIC_INHERITANCE). Parents need not exist yet; see E_UNDEFINED_PARENT.

#start_group [flavor] / #end_group [flavor]

Scoped configuration management and rollback boundaries. Supports optional bracketed options #start_group[callbacks=...].

#include [how] <data>

Slurps and processes input. For non-container modes (filename, string), strictly takes exactly 1 target argument. For container modes (section, configkey), strictly requires position 2 as the joiner key and position 3 as the target. Arity violations halt with E_SYNTAX_ERROR.

#include_scoped [how] [joiner] <data <flavor>>

Wraps inclusion in a group named flavor, guaranteeing complete rollback of configuration modifications upon EOF.

## [comment]

Comment line recognized by _default_comment_recognizer. Fuses preceding and succeeding text chunks across the comment boundary.

PRIMITIVE API REFERENCE

These are the low-level methods on the Text::Fab object that user-defined Parsers can call. The user-facing directives (like #set) are defined by the Parser, whose role is to convert these directives into calls to the Text::Fab methods listed below.

Output Control Primitives

out__target_section( $basename, $namespace ): Redirects subsequent output to the specified section, pushing it onto the target stack (see "Output"). Creates the section if it does not exist. The directive #target_section calls it.
out__end_section(): Pops the current target from the target stack, restoring the enclosing one. On an empty stack, and on the base entry of postponed output, it does nothing. The directive #end_section calls it.
out__flush(): Prints the postponed output, the section of the base entry of postponed output (_main:body), assembled, to Fab/output_handle, and empties that section. Does nothing if output was never postponed. See "Output".
out__create_section( $basename, $namespace ): Creates a section, clearing its content if it already exists.
out__embed( $recipe_basename, $recipe_ns, \@arg_sections, \%options ): Places an embed placeholder/promise into the current target section. \@arg_sections are positional operands mapped to $ARGVn. The %options hash can contain features like aim, with_blinder, or callbacks (the last one [Planned] {???}, see #emb). An undefined $recipe_ns is kept as such: the recipe is resolved late, during assembly, as described for #emb; so are the operands.
out__get_joined( $spec, \@joiners, \%options ): A utility for converting a section (representable as a list of text-parts and embed-promises) into a single string by joining text chunks and filtering embeds.

Configuration Primitives

cfg__set( $key, @values ): Sets the value of a key. For a scalar key, accepts at most one value (empty @values sets undef, exactly one value sets the scalar). Passing more than one value halts with E_TYPE_MISMATCH. For a list key, @values sets the list elements (or accepts a single array reference). For a hash key, @values accepts an even-sized list of key-value pairs or a single hash reference; an odd number of items halts with E_TYPE_MISMATCH.
cfg__append( $key, @values ): Appends one or more items to a list key, appends a string value to a scalar key, or merges an even-sized list of key-value pairs (or a hash reference) into a hash key. Appending an odd number of items to a hash key halts with E_TYPE_MISMATCH.
cfg__prepend( $key, @values ): Prepends one or more items to a list key (preserving their order at the front of the list), or prepends a string value to a scalar key. Calling cfg__prepend on a hash key is prohibited and halts with E_TYPE_MISMATCH.
cfg__get( $key, [$offset] ): When $offset is omitted, returns the scalar value, array reference, or hash reference. When $offset is provided, acts as an array index for list keys, or as a sub-key for hash keys. Passing $offset on a scalar key halts with E_TYPE_MISMATCH.
cfg__get_joined( $key, \@joiners, \%options ): A powerful utility that retrieves a list-type or hash-type key and joins its elements into a single string. The @joiners array provides one or more separators that are cycled through when joining. In particular, for hashes, joiner 0 is between the first key and value, the second separates the first and the second key/value pair etc.; hence with a single joiner, it is also used inside and between pairs. The %options can control complex formatting, for example specify permitted numbers of joiners modulo scalar @joiners. (E.g., if “joiners come in pairs”, then the number of emitted joiners must be even!) This is the option permitted_joiners_modulo: a list of the permitted remainders (or one remainder), each in 0 .. scalar(@joiners)-1; for the pairs above, permitted_joiners_modulo => [0]. For a hash the rule applies to the joiners between the pairs. A violation halts with E_TYPE_MISMATCH.
cfg__pop( $key, [@args] ): Pops count elements from a list key (defaulting to 1), or deletes one or more subkeys from a hash value when keys are passed. For a list key, count must be a non-negative integer (0 pops nothing); otherwise halts with E_TYPE_MISMATCH. Popping more elements than the list holds halts with E_LONG_POP.
cfg__prepend_elt( $key, $offset, $value ): Inserts $value at array index $offset for list keys, or inserts/updates a key-value pair [$k, $v] for hash keys. (The parser for directive may implement only the flavors with $offset in 0 and -1 only, e.g., “as if” in cfg__prepend_last( $key, $value ) etc. {???})

Working with Hashes

In addition to scalars and lists, the framework supports hashes as a first-class configuration data type. This is controlled by two special configuration keys:

Fab/hash_keys

Analogous to Fab/list_keys, this is a hash whose keys are the names of configuration variables that should be treated as hashes by the primitive operations.

# In a config file (registering a hash name):
#append Fab/hash_keys my_parameters 1

Fab/hash_key_sort_order

This key holds a subroutine reference that defines how the keys of a hash should be sorted when the hash needs to be treated as an ordered list (e.g., for cfg__get_joined). The subroutine receives two keys as arguments. The default is standard string comparison.

# In Perl code, to set numeric sorting:
$fab->cfg__set('Fab/hash_key_sort_order', sub { $_[0] <=> $_[1] });

Fab/max_recursion_depth

An integer setting the maximum permitted depth of the active embed stack during assemble(). Defaults to 1000. If recursive embeds exceed this ceiling, expansion halts with an E_CIRCULAR_EMBED exception.

Grouping and Postponement Primitives

group__start( [$flavor_name, \%options] )

Starts a scoped group. Which configuration keys group__end() restores depends on the flavor (full or partial undo); see #start_group in "Undoable Configuration Changes"" in "Grouping: Scoped.

group__end( [$flavor_name] )

Ends a scoped group.

group__postpone( $depth, $priority, $how, $method_name, @args )

Schedules a Text::Fab method call (or code reference) to run when the group at relative depth $depth is closed. All postponed actions have strict undo semantics: failure in one action does not abort the remaining undo actions or snapshot rollback. $priority is a real number: Priorities < 0 execute PRE-undo while the group scope is still active. Priority 0.0 is the snapshot rollback point where configuration is restored. Priorities >= 0 execute POST-undo in the restored enclosing scope. $how must be 'append' or 'prepend' to define execution order among actions with equal priority.

The closing frame remains active on the stack during unwinding, so actions can target the closing group at depth 0 or parent groups at depth > 0. Insertions into existing current/future priority buckets are consumed directly by the active loop without recalculation. Insertions into the past (time-machine actions) or into non-existing priorities trigger on-demand recalculation of the priority array.

(The parser for directives may allow more user-friendly UI, such as specifying “a path” to this group, as in “the last-but-one x group before the last two y groups.)

priority_for_file_close( [$dist_from_end] )

Calculates file-closing priority based on distance from the end: distance 0 (last closed file) is -1000.0, distance 999 is -1999.0, and distance >= 1000 is -2000.0 (overflow).

Input and Inclusion Primitives

include__text( $how, [$joiner_key], $source, [\%opts] )

$fab->include__text('filename',  $path,               [\%opts]);
$fab->include__text('string',    $content,            [\%opts]);
$fab->include__text('section',   $joiner_key, $spec,  [\%opts]);
$fab->include__text('configkey', $joiner_key, $key,   [\%opts]);

Slurps or expands content according to $how. An optional trailing hash reference \%opts is accepted across all modes. It holds the joining options, the same as the %options of cfg__get_joined() (such as permitted_joiners_modulo), and is passed on to the joining step in the 'section' and 'configkey' modes; in the 'filename' and 'string' modes it is accepted and ignored. The directive #include[...] passes its bracketed options here.

Processes included input. If $how is 'filename', $data is read from disk and processed. Recursion is tracked on an inclusion stack and mirrored in the configuration key opened_includes; attempting to include a file currently in progress halts with E_CYCLIC_INCLUDE. An unreadable file halts with E_FILE_READ_ERROR. If $how is 'string', $data is processed directly as an in-memory string.

If $how is 'section' or 'configkey', requires $joiner_key (which is ignored when reading from a scalar-valued key); the $source is interpreted as a section-name or a key, correspondingly. This retrieves the list of joiner from configuration and joins the content using cyclic separators before processing.

Namespace and Inheritance Primitives

NS__set_parents( $namespace, @parents ): Registers the list of immediate parent namespaces for $namespace in the inheritance DAG (stored under Fab/inheritance_graph/$namespace/parents). Before writing, the inheritance graph is validated for cycles; adding a parent that would create a directed cycle halts with E_CYCLIC_INHERITANCE.
assemble( $root_spec ): Assembles and expands the section specified by $root_spec (e.g. Namespace:Basename or bare Basename defaulting to namespace _main). Coordinates lens affection massaging, positional argument binding ($ARGV0...), and blinder masks.

Configuration Prefix and Call Stack API

For recursive processing, the Configuration contains a stack of prefixes at Fab/call_stack_prefixes. Primitives are provided to safely interact with keys relative to this stack.

(We list only the most basic retrieval primitives; the rest follows the same principle.)

call_stack__push( $prefix, [$filter_name] ): Pushes a new configuration prefix onto the call stack at Fab/call_stack_prefixes. If $filter_name is provided, pushes it onto Fab/call_stack_filter_name.
call_stack__pop(): Pops the topmost prefix from Fab/call_stack_prefixes and pops Fab/call_stack_filter_name. Guards the root prefix 'Fab/' from being popped, halting with E_UPLEVEL_TOO_DEEP on underflow.
cfg__get_prefixed( $key ): Retrieves the value of $key using only the current, most recent prefix (i.e., CURRENT_PREFIX/$key).
cfg__get_prefixed_scan( $key ): Retrieves the value of $key by searching for it under each prefix in the call stack, from most recent to least recent.
cfg__get_uplevel_scan( $level, $key ): Retrieves the value of $key by first popping $level prefixes from the call stack and then performing a prefixed search.
cfg__set_prefixed( $key, @values ): Sets the value of $key using the current (most recent) prefix.

For setting things “uplevel on this stack”, there are two methods matching two types of access:

cfg__set_prefixed_uplevel( $level, $key, @values ): As cfg__set_prefixed(), but goes back $level steps on the stack of prefixes. This matches retrieval via cfg__get_prefixed().
cfg__set_prefixed_uplevel_fixup( $level, $key, @values ): As cfg__set_prefixed_uplevel(), but first runs “fixup”: copying the preceding value at $level to the levels between this one and the current level — stopping when value is found on a particular level. This matches retrieval via cfg__get_prefixed(): this call won’t change the results of these calls on the levels between $level and the current one.

ADVANCED TOPICS

Input Processing and Recursion

The primitive to support the #include directive is include__text($how, @args). If $how is filename, the file $data is searched for and slurped; if not, $data is the input string to be processed. A special form #include_scoped ensures that configuration changes made by the included file are temporary by wrapping the call in a group named by the last argument. Active inclusion paths are mirrored in opened_includes.

Recursive processing (e.g., using one Text::Fab “process” to filter data for another) is managed entirely within the Configuration hash. A stack of configuration prefixes is maintained in Fab/call_stack_prefixes (which usually starts as just Fab/). When a named recursive filter is invoked, its name is pushed to the filter name stack Fab/call_stack_filter_name. This also pushes a certain prefix onto Fab/call_stack_prefixes via call_stack__push(). Since Text::Fab looks up its own settings under the prefixes on this stack, the most recent one first (see "The Configuration Hash" for the list of these settings), this allows a special predefined configuration (e.g., Fab/myList2ScalarFilter) to take precedence without conflicting with the “normal” operation of the Fab.

The uplevel family of configuration directives [Planned] {???} operates with respect to both the group stack and this call stack, allowing for powerful, controlled communication between different layers of processing. (The default parser has no such directives yet; currently only the primitives exist: cfg__get_uplevel_scan(), cfg__set_prefixed_uplevel(), cfg__set_prefixed_uplevel_fixup() and group__postpone().)

Namespace Resolution and Inheritance of output sections

The inheritance system is a layer of logic built on top of the primitive operations. It is controlled entirely by the Configuration hash. Here we illustrate the API by how the corresponding directives may look like. The API has an extra prefix NS__, and omits the UI verbiage.

#set_parents <namespace> <parent1> <parent2> ...: A user-facing directive that calls cfg__set on a key within the Configuration that stores the inheritance DAG. For example, it might set Fab/inheritance_graph/MyNamespace/parents to [Parent1, Parent2].

Resolution order. A section NS:name is looked up in NS first, then in the parents of NS depth-first, in the order in which they were declared; a namespace reached a second time (through another path of the DAG) is not searched again. In other words, a parent A is treated as a mix of A proper with whatever defaults A overrides, and this whole mix comes before the next parent. So with Page having parents (A, B), and both A and B having the parent Base, the search order is Page, A, Base, B: a section defined both in Base and in B is taken from Base. (A breadth-first order, Page, A, B, Base, may later be offered as an option. {???})

Parents may be declared before they exist. If the search reaches a parent which has neither sections nor parents of its own, assembly halts with E_UNDEFINED_PARENT.

The Assembly Phase uses this graph to resolve embedded placeholders/promises in the output sections; below we denote them as #emb. The resolution can be controlled with extra data:

#emb[aim=<Aim>, ...] <recipe> [<arg_sections...>]

This performs lens-guided resolution. The requested aim is passed through _massage_lens across the active lens stack via static joining rules and registered affections in Fab/lens_affects, establishing the virtual root in the DAG. When the virtual root is not the root, this makes the name resolution "blind" to any “override sections” up the tree/DAG. (Recall that the root contains the most specific overrides.)

The resolved operands are accessible as $ARGV0, $ARGV1, etc. in the recipe body. Since such a promise switches the lens, every operand must be tagged old=... or new=... (see #emb in "DEFAULT (dbg0) DIRECTIVES REFERENCE"). {???}

#emb[with_blinder=<SubName>, ...] <recipe> ...

Here SubName provides a completely customizable analog of the previous call, looked up as a subroutine reference under Fab/blinders/$SubName. Instead of (or in addition to) obscuring the part of DAG which is not reachable from the aim, the output of this user-defined subroutine obscures an arbitrary subset of the DAG. Blinding a namespace masks that namespace's own section definitions from satisfying the embed, while allowing inheritance traversal to continue through its parents in the DAG. It receives the entire current embedding stack (each entry is the currently executed embedding promise) together with the start namespace, the used blinder name, and the output of the blinder for each level. It returns a dynamic list of namespaces to "blind" for this specific resolution and any sub-resolutions (which may happen during processing of this embedded section). A name not registered under Fab/blinders/ halts with E_UNKNOWN_BLINDER.

Embed Cycle Detection and Recursion [NEEDED ???]

During assembly, Text::Fab detects circular embeds along the active resolution branch. For unparameterized sections, re-entering the same namespace:basename triggers an E_CIRCULAR_EMBED exception.

For parameterized recipes (invoked with argument sections), re-entry is permitted provided the active argument frame differs. This enables recursive recipe decomposition and iterative processing, bounded by Fab/max_recursion_depth.

The argument frame consists of the bound operands: each operand is turned into the namespace:basename where its resolution starts when the promise is expanded, and an operand $ARGVn is aliased to the caller's own n-th operand. So a recipe may pass its aliased operands on (with $ARGV3 becoming $ARGV0 inside inner:

#target_section outer in L
#emb inner $ARGV3
#end_section

Support of Control-Flow-Like constructs (conditionals and/or loops) [Planned] {???}

To simplify support of matched nested control-constructs in the user-defined parser, Fab maintains 6 stacks of matched lengths. Because of this match, they should be modified only via the corresponding API. E.g., the last elements of these lists describe:

The type of the last encountered starter of the construct. (Set by control__start())
The (optional) text label for the construct. (Likewise. Defined for “loop-like” constructs only.)
Are we in a skipping or not (“live”) branch/flavor of the construct. (With the value 2 if the processing were skipping even before meeting the start of the construct. Set likewise at start, then may be changed.)
The offset of the start of the body (for “loop-like” constructs only). (This may be the file offset for seek()able files, otherwise offset in a suitable buffer.)
Other useful offsets (for support of various loop-related targets for analogues of goto) packed into a hash.
The loop_counter (defined for “loop-like” constructs only). Changed (in a parser-specified ways) on “jump” calls.
Which targetToSeek we are trying to find in the “skipping” mode now.

There is also the list of the indices Fab/control__loop_indices of “loop-like” constructs in this stack.

Essentially, when encountering the start of an analogue of if/elsif/else/endif, the parser may registers its type, as well as whether the start is in the “skipping” mode via control__start(TYPE,IS_skipping,LABEL). Likewise, every flip of the skipping mode is registered with control__set_skipping(IS_skipping) (this is going to be ignored if the current skipping state is 2; likewise, Fab knows when to set the state 2 automatically at start). When the end of the construct is encountered, these data may be popped by control__end(TYPE) (with TYPE given for error-checking only).

The latest “skipping” state determines whether the preprocessor calls are skipped, and it is also given as an argument to the de-interlacer and parser callbacks. It also affects the state of loops nested inside “skipped” input.

These Control Stacks are reset when a new input file is included and restored when processing of that file completes.

The API to deal with these stacks is:

control__start( $type, $is_skipping, $label) [Planned] {???}

Pushes a new frame onto the Control Stack.

$type and $label are stored directly.
The initial skipping_state for the new frame is computed as follows: if the parent frame (if one exists) is already in any skipping state, the new frame's state is set to "skipping-on-entry" (state 2). Otherwise, the new frame's skipping state is set based on the boolean $is_skipping (true -> "skipping", false -> "live").

control__set_skipping( 'flip' | $boolean ) [Planned] {???}

Modifies the skipping_state of the current control block.

If the argument is 'flip', the state toggles between "live" and "skipping", but only if the current state is not "skipping-on-entry" (state 2).
If a boolean is provided, the state is set accordingly, again respecting the immutability of the "skipping-on-entry" state. This is called by the Parser for constructs like #elif.

control__end( $expected_type ) [Planned] {???}

Pops the top frame from the Control Stack. For error-checking, the Parser must provide the $expected_type. (Typically, this is used for loop-like constructs only when they are in “skipping mode”; see the next API.) Fab/control__loop_indices is changed accordingly.

control__define_invalidation_pointer($expected_type) [Planned] {???}

Mark the next position as “First postion not in the loop”. When this position “is activated”, or the jump is performed past this position (or before the starting position), the effect should be the same as control__end( $expected_type ).

control__set_pointer( $pointer_name, [$before], [$force] ) [Planned] {???}

Records the current position in the input stream (file or buffer) (if $before, this is the position before the directive) and stores it under $pointer_name in the "Custom Pointers" hash of the current control block. This is the sole mechanism for defining loop start points, continue targets, or any other labeled position. For all loop-like constructs, the Parser is expected to call this at the beginning of the loop body to define a 'body_start' pointer.

These pointers form the structural geometry of the loop. One must keep in mind that Fab may assume that this geometry is not modified (only enhanced by definition of new pointers) during “each pass” of a particular “instance of processing” this loop. (Here our terminology is a bit dense. If the current loop is nested inside another loop, it can be entered many times — leading to different instances of processing this loop. In each instance the body may be looped over in several passes.)

By default, this condition is auto-enforced: this call is ignored on the “replay passes” of “loop-like” constructs (i.e., if the current portion of the loop’s body “has already been read” — inside the current instance). This filtering may be disabled by the $force flag. (A similar flag $is_replay is passed to the parser — just in case.)

control__jump_to( $pointer_name, [$block_label] , [$incr_loop_counter]) [Planned] {???}

This is the primitive for all control flow transfers. It instructs the Fab core's main processing loop to change its read position.

If $block_label is given, it walks up the Control Stack to find the frame with that primary label. If not, it uses the current (top-most) frame.
It looks up $pointer_name in that frame's pointers hash.
If found, it commands the main loop to immediately stop scanning at the current location and resume reading from the retrieved pointer's location (either by seeking in a file or switching to a buffer at a specific offset).
If not found, it adds a new targetToSeek name, sets the current execution state to "skipping" (this essentially initiates the forward scan).

The loop_counter is incremented by the number in $incr_loop_counter (when the target is found).

Sometimes a callback may need a computationally expensive combination of individual entries from a configuration hash. (On may think of a regular expression depending on several values in this hash.) To avoid recalculating it every time, the callback may use the logic like this:

unless ($cache_at_count == (my $new_cnt = $fab->{config}{cnt__parser_REx})) {
  $cache = my_recalc_cache($Fab);
  $cache_at_count = $new_cnt;
}

This assumes that $fab->{config}{cnt__parser_REx} changes every time one of the variables used by my_recalc_cache() is updated (or restored). To support this, one register each of these variables with the “dependent counter” cnt__parser_REx using the API:

config__append_dependent_counter($counter_key, $KEY1, ...) [Planned] {???}

(Warning: the implementation is free to change the counter back when a group is undone if it knows all the relevant-and-changed variables were properly undone. So (theoretically) one cannot rely on the counter to increase-on-modification.

A tacit assumption — caveat usor

Summary: Before starting a design using Text::Fab, check that inside your preprocessor, the internal representation of the partially processed input can be easily enough kept in the Configuration Hash. We assume that this is going to hold as far as the preprocessor “proceeds with a well-defined processing pipeline”.

---

In many cases one needs to allow the configuration driving the logic of work of a preprocessor to be changeable by a directive at any time. (For example, a text in one programming language may inline a text in another programming language!)

The design of Fab’s API can make this (seem to be?) hard: (due it its up→down approach to the search of directives) the directive above would interrupt the input to the preprocessor; — but sometimes the preprocessor may need a non-local view on the input to decide how to deal with it. “Just seeing what comes before the directive”, then “Just seeing what comes after the directive” (in a separate invocation!) may be not enough for such a non-local view.

On the other hand, one should be ready for the situation when the input after this directive should be handled differently than the input before this directive. So if the target domain requires handling such situations, then any implementation of such a preprocessor must be ready to deal with such hiccups when processing the input (either by handling the update of its config, or bulking out with an error message). While doing this with a bottom→up design may seem easier, the experience shows that often it is prohibitively hard to implement. (E.g., when nested constructions are dealt with by recursive calls, the calls to reconfigure_me() would happen deep inside the stack of recursive calls!)

In short: to implement such a design through Text::Fab the only possible complication is: how to preserve the internal representation of the “preceding not-yet-fully-digested input” within the preprocessor “when the preprocessor is reentered” after handling such a directive?

With the bottom→up approach (when the preprocessor may “call some API” to handle the directive) this is easy: it is easy to make sure that such a call may update “the configuration” of the preprocessor while keeping “the internal representation of the already-read-input-to-preprocess” fully intact. However, when the preprocessor is controlled from a Fab, what happens instead of “calling the API” is: “exit the preprocessor”, then somebody else (the Fab!) calling the API, then “reenter the preprocessor again”. Therefore when the preprocessor can see that the chunk it obtained is a part of “some larger construct” (so it is not ready to be flush()'ed to the output section yet), it must preserve its internal state “somewhere”, then read it back when it is reentered (if it is ever reentered). This is the price to pay for (a lot of) work offloaded to Fab.

CONCLUSION:

In such situations the preprocessor needs a guarantee that it is going to be reentered;
Its internal data should be easily translatable to/from formats supported by the configuration hash.

(Indeed, this hash is the only supported storage provided by Fab’s API.)

In other words: the preprocessor cannot be stateless now, but as far as the state may be (de)serialized easily, it is not a big deal. So one possible tacit assumption about designs using Text::Fab may be:

The preprocessing is hierarchical, each level of hierarchy of not-yet-fully-processed-input may be put into a value suitable for the Configuration Hash, and the steps remaining to finish processing these data are not affected by the directive above.

————————————————————————————————————

Next, inspect how not-stateless but still “easy to implement” preprocessors may look like.

Example B: We have a text which may contain nested “blocks”, each block made of one or more paragraphs. The nesting is defined by the indent level (as in Python). The “class” of each block is determined by inspecting the immediate start of the its first paragraph, as well as inspecting the immediate end of the last paragraph of block (provided this last paragraph is “on the same indent level as the start” — so it is not in a nested subblock).

In addition to this we allow #define directives which take a whole line — but are not considered as interrupting the logic of indentation-which-defines-nested-depth.

Implementation: To implement “reenterancy”, the stored data consists of the stack of already-preprocessed text of non-yet-finished blocks. When preprocessing a chunk of input finishes a block, the preprocessor can inspect its start and end, determine its class, pop()'s the known “intermediate textural representation” from the stack, and converts it to the final representation in the way suitable for this class. When this was the outermost block, the result is put into the output section. Otherwise it is appended to the preceding entry on the stack (the containing block).

————————————————————————————————————

Example H: As above, but the rules for massaging a block depend not only on its class, but also on the classes of its parents. This requires “the most general” hierarchical model of the pipeline of not-yet-fully-processed input.

In this case again the handled data may be flush()'ed to an output section only when the outermost encountered block is terminated. Until then, one should maintain the stack of “live blocks” (not yet terminated), as well as a forest of “dead” (already terminated, but waiting the finite processing) blocks. Each terminated block has its class and the list of content, elements are either “a literal text” (preprocessed-except-the-final-step), or another (nested) terminated sub-block. The live blocks are likewise, but the class is not yet fully known, and the last nested sub-block may be live.

How to preserve this in the Configuration Hash? We have a planar rooted tree with vertices assigned the class (or undef for live vertices), and leaves containing “a plain text”. When flush()'ing to an output section, we essentially scan depth-first through the vertices, maintaining the stack of classes (they correspond to the path from the root to the current vertex). So the only bookkeeping data in addition to the stack of classes is the (parallel) stack of “where the currently-not-yet-finished sub-block ends”. (So we can pop() the stack of classes when this end is reached.)

So if we store the leaves in the depth-first order, we need to interlace this list with markers “start sub-block of the class CLASS ending at position POS” (or introduce nested start and end markers). To design an easy representation, suppose that the semantic of “massaging the blocks” allows introdution of empty "plain text leaves" between every two adjacent sub-vertices; then we may assume that “end-subblock” entries are represented by out-of-bound elements (e.g., undef), and the plain text leaves are alternating with “start subblock” entries in the stored list, as in flattened list of the following form:

0thLlvTxt1
  Blk1
    1stLvl1Txt11
      Blk11
        2ndLvl1Txt111
      undef
    1stLvl1Txt12
      Blk12
        2ndLvl1Txt121
      undef
    1stLvl1Txt13
  undef
0thLlvTxt2
  Blk2
    1stLvl1Txt21
      Blk21
        2ndLvl1Txt211
      undef
    1stLvl1Txt22
      Blk22
        2ndLvl1Txt221
      undef
    1stLvl1Txt23
  undef
0thLlvTxt3

Observe alternation Txt/BlkClass/Txt/BlkClass/Txt/… after every undef. It is trivial to process such a list when flush()'ing. When filling the list, one should additionally maintain a stack of positions in this list where live block starts, so one can change a placeholder-for-its-class to the calculated class when the end-of-block is found. (Other than this, all the results of pre-processing the input land at the end of the list.)

————————————————————————————————————

Example A: Consider a macroprocessor for macros with arguments; assume that the argument-separators and argument-terminators for a macro on a certain level of nesting may be contained in the output of nested-deeper macros. So one needs to expand the deeper macros before the handling of enclosing macros is finished. Can one support directives occuring deep inside the nested arguments — but we assume they cannot be contained in the macro-expansions?

Here the data may be even easier than in the preceding example. Since the output produced up to the start of the outermost no-yet-terminated macro may be flash()'ed, the stack of unprocessed data may contain the ID of the outermost no-yet-terminated macro, the list of its completed arguments, then the already expanded prefix of the current argument, then the ID of the next no-yet-terminated macro, the list of its completed arguments etc. In addition to this list, it is enough to have the list of offsets at which the IDs of the macros live.

When a terminator of a macro is found, one pops from the list above the macro ID and the completed arguments, and performs the macrosubstitution. There are two ways to proceed with this string: it is either appended to “the already expanded prefix of the current argument” (as above) of the enclosed macro (if the expansion should not be macro-re-expanded) — and optionally scanned for macro-argument-delimiter or -terminator, or it is prepended to the buffer containing the not-yet-processed input (otherwise). The desired choice is determined by the semantic of the macro-expansion.

————————————————————————————————————

Example E: As above, but the directives may appear as the result of macro-expansion.

This case is much more involved, since it seems it is too late for the Fab to inspect the internal state of the macro-processor. So the preprocessor should scan its output itself: it has the full access to the De-Interlacer/Parser/etc callbacks designed to detect the directives.

WARNING: However, the directives are not designed to work with not-yet-completed buffers. For best result, De-Interlacer should return “the high-water mark”: the offset before which the directives cannot appear — so there is no sense to rescan this part when the buffer is extended.

ERROR REFERENCE

The preprocessor will halt with one of the following error codes if a fatal condition is met.

Parsing and Input Errors (`E_Input`)

E_SYNTAX_ERROR: A directive does not conform to the grammar, or an unrecognized directive is encountered.
E_FILE_READ_ERROR: An #included file cannot be opened or read.
E_CYCLIC_INCLUDE: An #include chain references a file that is already being processed.

Grouping Errors (`E_Group`)

E_MISMATCHED_END_GROUP: An #end_group <flavor> does not match the currently open group.
E_DANGLING_END_GROUP: An #end_group is found with no matching #start_group.
E_INVALID_GROUP_NESTING: An attempt to nest groups in a way forbidden by the Configuration.
E_UPLEVEL_TOO_DEEP: An uplevel primitive (or, [Planned] {???}, an uplevel directive), a call stack pop, or group__postpone() targets a level that does not exist.

Configuration Errors (`E_Config`)

E_TYPE_MISMATCH: A list-specific operation is attempted on a scalar key, or vice-versa. Also used for wrong numbers or kinds of arguments to primitives (for example an odd number of items for a hash key, an invalid cfg__pop count), and when a join violates permitted_joiners_modulo.
E_LONG_POP: Popping too many elements.

Namespace and Assembly Errors (`E_Assembly`)

E_CYCLIC_INHERITANCE: An #set_parents directive creates a loop in the inheritance DAG.
E_UNDEFINED_PARENT: During assembly, the resolution reaches a parent namespace (declared with #set_parents) which has neither sections nor parents of its own. (#set_parents itself accepts parents which do not exist yet.)
E_ROOT_NOT_FOUND [Planned]: A section specified via #set_root [Planned] does not exist. (Currently assemble() takes the root section as its argument.)
E_NO_ROOTS_SPECIFIED [Planned]: The assembly phase is triggered, but no roots were ever defined with #set_root [Planned].
E_EMBED_NOT_FOUND: A section referenced by an #emb placeholder/promise cannot be resolved.
E_PROMISE_IN_DIRECT_OUTPUT: An #emb is met while the output goes directly to Fab/output_handle, and Fab/output_postpone is error (see "Output").
E_CIRCULAR_EMBED: An #emb chain results in a loop during the final assembly.
E_UNKNOWN_BUNDLE: A requested callback bundle is not registered under Fab/callbacks_bundle/.
E_ARGV_EMPTY: Referenced an $ARGVn section parameter outside of a recipe invocation.
E_ARGV_BOUNDS: Referenced an $ARGVn index that exceeds the number of supplied section arguments.
E_UNKNOWN_BLINDER: A promise names a blinder (with_blinder=...) which is not registered under Fab/blinders/.
E_UNTAGGED_OPERAND: A promise that switches the lens (aim=...) has an operand not tagged old= or new=.

Control Flow Errors (`E_Control`) [Planned] {???}

E_MISMATCHED_CONTROL_END: An end directive (e.g., #endif) was encountered, but the currently open control block is of a different type (e.g., a #for loop).
E_DANGLING_CONTROL_END: An end directive was encountered when the Control Stack was empty.
E_UNCLOSED_CONTROL_BLOCK: The end of an input file was reached while one or more control blocks were still open.
E_POINTER_NOT_FOUND: A control__jump_to call referenced a pointer name that has not been defined in the target control block.
E_BLOCK_LABEL_NOT_FOUND: A control__jump_to call referenced a block label that does not exist on the Control Stack, and was not found before the end of the enclosing block.

Payload: (target_pointer_name, block_label)

PLACEHOLDERS

Callback signatures, as the code calls them now. {???}

The active callbacks are the hash in Fab/callbacks (or, if it holds a name, the bundle Fab/callbacks_bundle/NAME); the default bundle is dbg0. They are fetched once at the start of each process_string().

interleaver( $buffer, $offset ): Returns the offset of the next directive at or after $offset, or undef if there is none. (It does not get $fab.) dbg0: a # at the start of a line.
comment_recognizer( $fab, $buffer, $offset ): Called with the offset found by the interleaver. Returns the offset after the comment if the directive there is a comment, else undef. The text before a comment is not flushed; it is fused with the text after it. dbg0: a line starting with ##.
directive_parser( $fab, $buffer, $offset ): Executes the directive at $offset (usually by calling primitive methods) and returns the offset where scanning resumes. dbg0: the directives in "DEFAULT (dbg0) DIRECTIVES REFERENCE".
chunk_preprocessor( $fab, $chunk ): Returns the processed text of a chunk (the text between directives, with comments fused); Text::Fab appends it to the current target section. dbg0: returns $chunk unchanged.
error_handler( $fab, $type, \%details ): Called by dbg0's parser with $type 'unknown_directive' (%details: directive, line) or 'malformed_directive' (line). dbg0: dies with E_SYNTAX_ERROR.
Blinders: Fab/blinders/NAME->( $fab, \%info ): %info has embed_stack (the frames of the enclosing promises, each with section, options and blinder_output), start_namespace, blinder_name and target_section. Returns the list of namespaces to blind.

The text below is the earlier plan for these sections. {???}

Future sections will detail the full APIs and default implementations for:

The Parser: How it interacts with the main loop and calls primitives.
Chunk Preprocessors: The interface for the "stomach".
Blinder Subroutines: The data structures passed to them.
The default configuration schema under the Fab/ key:

Fab/callbacks_bundle/dbg0

A hash reference containing the canonical operational engine callbacks: interleaver, directive_parser, chunk_preprocessor, comment_recognizer, and error_handler. {??? The active callbacks bundle is referenced via Fab/callbacks.}

The comment_recognizer callback receives ($fab, $buffer, $offset) and returns the offset after the comment line if recognized (defaulting to lines prefixed with ##), which triggers chunk fusion across the comment boundary. {??? Docs of the rest?}

EXPORT

None by default.

TODO

Maybe use caching counters to automate conversion of `callbacks` from a string to a hash of subroutines???

Need a test which merges peano with infix, then one implementing all of 50-54 basing only on unary and Peano primitives. m4???

Document which parts of "Fab/" Config are accessed through an explicit path, and which through a dynamic one.

A conflict between installable components's names may arise; so they need to allow redefinition of the name/prefix they use.  Or:
they requestion "Foo", get back "Foo12" (so the requested name should not end in a digit).

Efficient replacement for FreezeThaw. Handle DAG config - but probably not an arbitrary graph with weak links inside??? Handle a tree and the knowledge of where is the gluing of branches???

How switching the filter interacts with dynamic switching of Fabs??? Where the switched filter stores its data: explicitly in subsets of config??? Interaction with dynamic parts of the config??? Compartmentalization of config???

How to indicate resolution in the caller/callee namespace??? What is the default now???

How to decide “executed vs. postponed” for promises: which promises are expanded now, and which "pass through" unchanged? Compare a “do-when” “watermark” set in the promise and in the piping stage?

Traceback of provenance of errors in presence of promises??? Traceback of who created the promise in its metadata???

User-configurable logging/debugging output???

For postponed actions need an extra argument "ignore errors": with 2 would just put into eval, with 1 converted to a warning, negative values stop the remaining undo actions for that many frames (with -2 stopping for the current frame and one more).

One needs to prohibit __get()ing a reference. Use __getref() when you expect a reference (but no checks for reference are performed).

MISS: include__text() reading from a named filehandle (e.g. STDIN), not only from a file name or a string.

MISS: detecting the end of a job. A job should be a group (the outermost one); ending the job closes all open groups down to it, running everything postponed (which may add output, promises or new groups) until nothing is left, and printing the postponed output is one of these postponed actions. There should be ways to declare that every job consists of one top-level include, or that a particular (top-level only???) include finishes the input when it ends.

MISS: input__end($n) with $n==0 ending the current input stream, or also $n enclosing streams.

MISS: need to specify what happens with a group on input__end(): survives / should be closed before / close-if-still-open.

MISS: input__end_to($group_type): end input of the files opened up to the the opened-latest-group of the given type.

Likewise for “closing enclosed groups”??? Join together into 1 API by introducing suitable groups???

  May run forever??? Even the language if group__start, group_end, group__postpone (with the arguments to the last command
  restricted to these 3 calls) is probably Turing complete???

MISS: _-prefixed versions of config__*() which check the prefix _ __ ___ of the key and make it read/write only.

MISS: pipelining preprocessor with “filters”: filters consists of 3 components: preprocessor, massaging of promises, and the concatenator of the resulting candidates-for-sections.

Uplevel w.r.t. the recursion stack should be recalculated into the usual uplevels; a special do-nothing group may be used.

allow not 1 active preprocessor engine, but a stack of them, chained, and have a cfg__pop() primitive method.

Macros are templates marked as OK-to-be-ignored.

Hash to mark macros as “types”, with per-type-warning configurable? Which way to hash, types to macros, or macros to types, or pairs???

An extra parameter $is_replay for the syntax callbacks for the non-first pass. (Probably, the boundary of the preceding pass cannot be part of this???

Hash values. Setting/getting them employs a serialization method, by default JSON???

     !!!!!!!!!!!!!!!!!!!!!!!!!!!!	THIS IS TOO DANGEROUS.
					Access control to Fab/* keys should be guarded by ungettable/settable types of values ???

#unset — but maybe it is going to make undo harder???

Recursive invocation is needed for massaging data between different levels (currently, these are convertable to output sections):

• The hash
• Input file names
• Input strings
• Output sections
• Arguments to directives/promises

A top→down approach to recognizing the “reconfiguration directives”: but cannot we support a bottom→up approach too, when the preprocessor recognizes interlaced directives (<b>x<i>y</b>z</i>? How to deal with recursion: return an indicator that the state changed? But it is changing permanently, one needs to filter out changes — but how the internal calls know the filters needed for the enclosing calls???

Resolving to _main namespace is broken (too early); it should be left undefined, so the resolver knows it should do the late resolution — at runtime, when the lenses may be set up differently. DONE: unqualified recipes and operands are now resolved during assembly.

the preprocessor needs a guarantee that it is going to be reentered

---

The default mode of processing is equivalent to writing to a buffer __default__, and at the end executing the file

= = = = = filterbuffer(raw) __default__

Doing the same abstraction once more, the buffer with that contents may be also created dynamically, then loaded via filterbuffer(). On the gripping hand, this may be also

= = = = = filterbuffer(raw) __default__pre
= = = = = filterbuffer(raw) __default__
= = = = = filterbuffer(raw) __default__post

With (initially) empty buffers __default__pre and __default__post (to which one has a chance to write later, during creation of __default__!). And what one writes to these buffers may be triplicated likewise. Etc.

AUTHOR

Ilya Zakharevich, <ilyaz@cpan.com>

COPYRIGHT AND LICENSE

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.22.2 or, at your option, any later version of Perl 5 you may have available.

To install Text::Fab, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Text::Fab

CPAN shell

perl -MCPAN -e shell
install Text::Fab

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	Go to GitHub issues (only if GitHub is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

SYNOPSIS

DESCRIPTION

Preview of architecture

Example of usage

Massaging of lens names

THE PROCESSING MODEL

The Configuration Hash

The Output Sections

Target Section Stacking and Scoping

Output

Grouping: Scoped/Undoable Configuration Changes

DEFAULT (dbg0) DIRECTIVES REFERENCE

PRIMITIVE API REFERENCE

Output Control Primitives

Configuration Primitives

Working with Hashes

Grouping and Postponement Primitives

Input and Inclusion Primitives

Namespace and Inheritance Primitives

Configuration Prefix and Call Stack API

ADVANCED TOPICS

Input Processing and Recursion

Namespace Resolution and Inheritance of output sections

Embed Cycle Detection and Recursion [NEEDED ???]

Support of Control-Flow-Like constructs (conditionals and/or loops) [Planned] {???}

Support of Caching configuration-related data [Planned] {???}

A tacit assumption — caveat usor

ERROR REFERENCE

Parsing and Input Errors (E_Input)

Grouping Errors (E_Group)

Configuration Errors (E_Config)

Namespace and Assembly Errors (E_Assembly)

Control Flow Errors (E_Control) [Planned] {???}

PLACEHOLDERS

EXPORT

TODO

SEE ALSO

AUTHOR

COPYRIGHT AND LICENSE

Module Install Instructions

Keyboard Shortcuts

DEFAULT (`dbg0`) DIRECTIVES REFERENCE

Parsing and Input Errors (`E_Input`)

Grouping Errors (`E_Group`)

Configuration Errors (`E_Config`)

Namespace and Assembly Errors (`E_Assembly`)

Control Flow Errors (`E_Control`) [Planned] {???}