HTML::Content::HTMLTokenizer - Perl module to tokenize HTML documents.
use HTML::Content::HTMLTokenizer; my $tokenizer = new HTML::Content::HTMLTokenizer('TAG','WORD'); open(HTML,"index.html"); my $doc = join("",<HTML>); close(HTML); my ($word_count_arr_ref,$tag_count_arr_ref,$token_type_arr_ref,$token_hash_ref) = $tokenizer->Tokenize($doc);
HTML::Content::HTMLTokenizer has one main method, Tokenize, which tokenizes a HTML document into a sequence of 'TAG' and 'WORD' tokens.
my $tokenizer = new HTML::Content::HTMLTokenizer($tagMarker,$wordMarker)
Initializes HTML::Content::HTMLTokenizer.
$tagMarker - String that will represent tags in the token sequence returned from Tokenize.
$wordMarker - String that will represent words in the token sequence returned from Tokenize.
my (\@WordCount,\@TokenCount,\@Sequence,\%Tokens) = $tokenizer->Tokenize(\$htmldocument);
$WordCount[$i] is the number of word tokens before or at the ith token in the input HTML document.
$TagCount[$i] is the number of tag tokens before or at the ith token in the input HTML document.
$Sequence[$i] is the type of token at the ith spot in the input HTML document. Either $tagMarker or $wordMarker.
$Tokens{$i} is the word at the ith spot in the input HTML document. This is defined only if there is a word at the ith spot in the document.
Jean Tavernier (jj.tavernier@gmail.com)
Copyright 2005 Jean Tavernier. All rights reserved.
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
ContentExtractorDriver.pl (1), HTML::Content::ContentExtractor (3), HTML::WordTagRatio::Ratio (3),HTML::WordTagRatio::WeightedRatio (3), HTML::WordTagRatio::SmoothedRatio (3), HTML::WordTagRatio::RelativeRatio (3), HTML::WordTagRatio::ExponentialRatio (3), HTML::WordTagRatio::NormalizedRatio (3).
To install HTML::WordTagRatio::Ratio, copy and paste the appropriate command in to your terminal.
cpanm
cpanm HTML::WordTagRatio::Ratio
CPAN shell
perl -MCPAN -e shell install HTML::WordTagRatio::Ratio
For more information on module installation, please visit the detailed CPAN module installation guide.