DTA::TokWrap::Processor::tcftokenize - DTA tokenizer wrappers: TCF text layer tokenization
use DTA::TokWrap::Processor::tcftokenize; $ttok = DTA::TokWrap::Processor::tcftokenize->new(%opts); $doc_or_undef = $ttok->tcftokenize($doc);
DTA::TokWrap::Processor::tcftokenize provides an object-oriented DTA::TokWrap::Processor wrapper for tokenizing the TCF text layer with the selected tokenizer and encoding the result in the TCF tokens and sentences layers.
text
tokens
sentences
DTA::TokWrap::Processor::tcftokenize inherits from DTA::TokWrap::Processor.
$obj = $CLASS_OR_OBJECT->new(%args);
Constructor.
%defaults = $CLASS->defaults();
Static class-dependent defaults.
$doc_or_undef = $CLASS_OR_OBJECT->tcftokenize($doc);
Tokenizes the text layer extracted from a TCF document and encodes the result in new TCF tokens and sentences layers.
Relevant %$doc keys:
tcfdoc => $tcfdoc, ##-- (input,output) TCF input document with <text> layer ## txtfile => $txtfile, ##-- (temp,output) text file used for TCF extraction tokdata0 => $tokdata, ##-- (temp,output) raw tokenization data tokdata1 => $tokdata1, ##-- (temp,output) tweaked tokenization data ## tcftokdoc => $tcftokdoc, ##-- (output) output TCF file with <sentences>,<tokens> layers (==$tcfdoc) tcftokenize_stamp0 => $f, ##-- (output) timestamp of operation begin tcftokenize_stamp => $f, ##-- (output) timestamp of operation end tcftokdoc_stamp => $f, ##-- (output) timestamp of operation end
DTA::TokWrap::Intro(3pm), dta-tokwrap.perl(1), ...
Bryan Jurish <moocow@cpan.org>
Copyright (C) 2014-2018 by Bryan Jurish
This package is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.14.2 or, at your option, any later version of Perl 5 you may have available.
To install DTA::TokWrap, copy and paste the appropriate command in to your terminal.
cpanm
cpanm DTA::TokWrap
CPAN shell
perl -MCPAN -e shell install DTA::TokWrap
For more information on module installation, please visit the detailed CPAN module installation guide.