Treex::Block::W2A::EN::Tokenize - rule-based tokenization
version 2.20151102
Each sentence is split into a sequence of tokens using a series of regexs. Flat a-tree is built and attributes no_space_after are filled. This class uses English specific regex rules (insipred by PennTB) for tokenization of contractions like He's, we'll, they've, don't etc.
no_space_after
David Mareček <marecek@ufal.mff.cuni.cz>
Martin Popel <popel@ufal.mff.cuni.cz>
Copyright © 2011 by Institute of Formal and Applied Linguistics, Charles University in Prague
This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install Treex::EN, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Treex::EN
CPAN shell
perl -MCPAN -e shell install Treex::EN
For more information on module installation, please visit the detailed CPAN module installation guide.