Treex::Block::W2A::ResegmentSentences - split bundles which contain more sentences
version 2.20151102
Some resources (most notably WMT newstest) are segmented to chunks of text which mostly correspond to sentences, but sometimes contain more than one sentence. Sometimes we want to process such documents in Treex and output (Write::*) the result in a format where one output segement correspond to one input segement. (So e.g. for "one-sentence-per-line writers", we have the same number of input and output lines.)
However, most Treex blocks expect exactly one (linguistic) sentence in each bundle. The solution is to use block W2A::ResegmentSentences after the reader and Misc::JoinBundles before the writer.
W2A::ResegmentSentences
Misc::JoinBundles
If the sentence segmenter says that the current sentence is actually composed of two or more sentences, then new bundles are inserted after the current bundle, each containing just one piece of the resegmented original sentence.
This block should be executed before tokenization (and tagging etc). It deals only with the (string) attribute sentence in each zone, it does not process any trees.
sentence
All zones are processed. The number of bundles created is determined by the number of subsegments in the "current" zone (specified by the parameters language and selector). If a zone contains less subsegments than the current one, the remaining bundles will contain empty sentence. If a zone contains more subsegments than the current one, the remaining subsegments will be joined in the last bundle.
language
selector
In other words, it is granted that the current zone, will not contain empty sentences.
As a special case if parameters language and selector define a zone which is not present in a bundle (this holds also for language=all), the "current" zone is the one with most subsegments, i.e. no subsegments are joined.
remove
Treex::Block::Misc::JoinBundles
Zdeněk Žabokrtský <zabokrtsky@ufal.mff.cuni.cz>
Martin Popel <popel@ufal.mff.cuni.cz>
Copyright © 2011-2012 by Institute of Formal and Applied Linguistics, Charles University in Prague
This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install Treex::Unilang, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Treex::Unilang
CPAN shell
perl -MCPAN -e shell install Treex::Unilang
For more information on module installation, please visit the detailed CPAN module installation guide.