Dušan Variš
and 1 contributors


Treex::Block::W2A::SegmentOnNewlines - segment text on new lines


version 2.20151102


The source text is segmented into sentences which are stored in document bundles. If the document contained no bundles, the bundles are created. Otherwise, the document must contain the same number of bundles as the number of sentences (segmented by this blocks). This means that this block (or its derivatives) can be used in this way:

    treex Read::Text language=en from=en.txt \
          Read::Text language=de from=de.txt \
          W2A::SegmentOnNewlines language=en \
          W2A::SegmentOnNewlines language=de \

This class detects sentences based on the newlines in the source text, but it can be used as an ancestor for more apropriate segmentations by overriding the method get_segments.



If set, empty sentences can be produced.


If set, empty sentences are automatically deleted.

If none of the previous attributes is set and empty sentence found, fatal is raised.


my @sentences = $self->get_segments($text)

This method produces a list of segments (sentences) from the given text string. This implementation only splits on newlines. It is supposed to be overriden in subclasses.

my $norm_sentence = $self->normalize_sentence($raw_sentence)

This method does sentence normalization, e.g. trims initial and terminal whitespaces.


Martin Popel <popel@ufal.mff.cuni.cz>


Copyright © 2011 by Institute of Formal and Applied Linguistics, Charles University in Prague

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.