Lingua::CJK::Tokenizer - CJK Tokenizer ++

This module tokenizes CJK texts into n-grams. METHODS ngram_size sets the size of returned n-grams max_token_count sets the limit on the number of returned n-grams in case input text is too long or of indefinite size tokenize tokenizes texts into n-g...

XERN/Lingua-CJK-Tokenizer-0.01 - 23 May 2009 18:51:02 GMT - Search in distribution