NAME

Treex::Block::W2A::Tokenize - language independent rule based tokenizer

VERSION

version 2.20151102

DESCRIPTION

Each sentence is split into a sequence of tokens using a series of regexs. Flat a-tree is built and attributes no_space_after are filled. This class uses language-independent regex rules for tokenization, but it can be used as an ancestor for language-specific tokenization by overriding the method tokenize_sentence or by using around (see Moose::Manual::MethodModifiers).

AUTHOR

David Mareček <marecek@ufal.mff.cuni.cz>

Martin Popel <popel@ufal.mff.cuni.cz>

COPYRIGHT AND LICENSE

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

To install Treex::Unilang, copy and paste the appropriate command in to your terminal.

cpanm

cpanm Treex::Unilang

CPAN shell

perl -MCPAN -e shell
install Treex::Unilang

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	go to github issues (only if github is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

	Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)

NAME

VERSION

DESCRIPTION

AUTHOR

COPYRIGHT AND LICENSE

Module Install Instructions