NAME

PDL::Ngrams::Utils - Basic N-Gram utilities for PDL: low-level utilities

SYNOPSIS

use PDL;
use PDL::Ngrams::Utils;

##---------------------------------------------------------------------
## ... stuff happens

FUNCTIONS

Delimiter Insertion and Removal

ng_delimit

Signature: (toks(NToks); indx boffsets(NBlocks); delims(NDelims); [o]dtoks(NDToks))

Add block-delimiters to a raw token vector

Splices the vector $delims into the vector $toks starting at each index listed in $boffsets, returning the result as $dtoks. Values in $boffsets should be in the range [0..N-1].

For consistency, it should be the case that:

$NDToks == $NToks + $NBlocks * $NDelims

ng_delimit does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

ng_undelimit

Signature: (dtoks(NDToks); indx boffsets(NBlocks); int NDelims(); [o]toks(NToks))

Remove block-delimiters from a delimited token vector.

Removes chunks of length $delims from the vector $toks starting at each index listed in $boffsets, returning the result as $toks. Values in $boffsets should be in the range [0..N-1].

ng_undelimit does not process bad values. It will set the bad-value flag of all output piddles if the flag is set for any of the input piddles.

ACKNOWLEDGEMENTS

Perl by Larry Wall.

PDL by Karl Glazebrook, Tuomas J. Lukka, Christian Soeller, and others.

KNOWN BUGS

Probably many.

AUTHOR

Bryan Jurish <moocow@cpan.org>

Copyright Policy

Copyright (C) 2007-2022, Bryan Jurish. All rights reserved.

This package is free software, and entirely without warranty. You may redistribute it and/or modify it under the same terms as Perl itself.

SEE ALSO

perl(1), PDL(3perl)