NAME

HTML::Parser - HTML tokenizer

SYNOPSIS

 require HTML::Parser;
 $p = HTML::Parser->new;  # should really a be subclass
 $p->parse($chunk1);
 $p->parse($chunk2);
 #...
 $p->eof;                 # signal end of document

 # Parse directly from file
 $p->parse_file("foo.html");
 # or
 open(F, "foo.html") || die;
 $p->parse_file(*F);

DESCRIPTION

The HTML::Parser will tokenize an HTML document when the parse() or parse_file() methods are called. Tokens are reported by invoking various callback methods. The document to be parsed can be supplied in arbitrary chunks.

The methods that make up the external interface of the HTML::Parser are:

$p = HTML::Parser->new

The object constructor takes no arguments.

$p->parse( $string );

Parse $string as the next chunk of the HTML document. The return value is a reference to the parser object (i.e. $p).

$p->eof

Signals the end of the HTML document. Calling the eof() method will flush any remaining buffered text. The return value is a reference to the parser object.

$p->parse_file( $file );

This method can be called to parse text directly from a file. The $file argument can be a filename or an already opened file handle (or a reference to such a handle).

If $file is a plain filename and the file can't be opened, then the method will return an undefined value and $! will tell you why it failed. In all other cases the return value will be a reference to the parser object.

If a filehandle is passed in, then the file will be read until EOF, but not otherwise affected.

$p->strict_comment( [$bool] )

By default we parse comments similar to how the popular browsers (like Netscape and MSIE) do it. This means that comments will always be terminated by the first occurrence of "-->". This is not correct according to the "official" HTML standards. The official behaviour can be enabled by calling the strict_comment() method with a TRUE argument.

The return value from strict_comment() is the old attribute value.

In order to make the parser do anything interesting, you must make a subclass where you override one or more of the following methods as appropriate:

$self->declaration($decl)

This method is called when a markup declaration has been recognized. For typical HTML documents, the only declaration you are likely to find is <!DOCTYPE ...>. The initial "<!" and ending ">" is not part of the string passed as argument. Comments are removed and entities will not be expanded.

$self->start($tag, $attr, $attrseq, $origtext)

This method is called when a complete start tag has been recognized. The first argument is the tag name (in lower case) and the second argument is a reference to a hash that contain all attributes found within the start tag. The attribute keys are converted to lower case. Entities found in the attribute values are already expanded. The third argument is a reference to an array with the lower case attribute keys in the original order. The fourth argument is the original HTML text.

$self->end($tag, $origtext)

This method is called when an end tag has been recognized. The first argument is the lower case tag name, the second the original HTML text of the tag.

$self->text($text)

This method is called when plain text in the document is recognized. The text is passed on unmodified and might contain multiple lines. Note that for efficiency reasons entities in the text are not expanded. You should call HTML::Entities::decode($text) before you process the text any further.

A sequence of text in the HTML document can be broken between several invocations of $self->text. The parser will make sure that it does not break a word or a sequence of spaces between two invocations of $self->text().

$self->comment($comment)

This method is called as comments are recognized. The leading and trailing "--" sequences have been stripped off the comment text.

The default implementation of these methods do nothing, i.e., the tokens are just ignored.

There is really nothing in the basic parser that is HTML specific, so it is likely that the parser can parse other kinds of SGML documents. SGML has many obscure features (not implemented by this module) that prevent us from renaming this module as SGML::Parser.

EFFICIENCY

The parser is fairly inefficient if the chunks passed to $p->parse() are too big. The reason is probably that perl ends up with a lot of character copying as tokens are chopped of from the beginning of the strings. A chunk size of about 256-512 bytes was optimal in a test I made with some real world HTML documents. (The parser was about 3 times slower with a chunk size of 20K).

SEE ALSO

HTML::Entities, HTML::TokeParser, HTML::Filter, HTML::HeadParser, HTML::LinkExtor

HTML::TreeBuilder (part of the HTML-Tree distribution)

COPYRIGHT

Copyright 1996-1999 Gisle Aas. All rights reserved.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.