The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.

Add one token to the batch.

Add many tokens to the batch, by supplying the string to be tokenized, and arrays of token starts and token ends (specified in bytes).

Proceed to the next item in the TokenBatch. Returns true if the TokenBatch ends up located at valid token.