The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.

Add one token to the batch.

Add many tokens to the batch, by supplying the string to be tokenized, and arrays of token starts and token ends (specified in bytes).

Proceed to the next item in the TokenBatch. Returns true if the TokenBatch ends up located at valid token.