NAME

HTML::TokeParser - Alternative HTML::Parser interface

SYNOPSIS

 require HTML::TokeParser;
 $p = HTML::TokeParser->new("index.html") || die "Can't open: $!";
 while (my $token = $p->get_token) {
     #...
 }

DESCRIPTION

The HTML::TokeParser is an alternative interface to the HTML::Parser class. It basically turns the HTML::Parser inside out. You associate a file (or any IO::Handle object or string) with the parser at construction time and then repeatedly call $parser->get_token to obtain the tags and text found in the parsed document. No need to make a subclass to make the parser do anything.

Calling the methods defined by the HTML::Parser base class will be confusing, so don't do that. Use the following methods instead:

$p = HTML::TokeParser->new( $file_or_doc );

The object constructor argument is either a file name, a file handle object, or the complete document to be parsed.

If the argument is a plain scalar, then it is taken as the name of a file to be opened and parsed. If the file can't be opened for reading, then the constructor will return an undefined value and $! will tell you why it failed.

If the argument is a reference to a plain scalar, then this scalar is taken to be the literal document to parse. Changing the value of this scalar before all tokens have been extracted is not allowed.

Otherwise the argument is taken to be some object that the HTML::TokeParser can read() from when it need more data. Typically it will be a filehandle of some kind.The stream will be read() until EOF, but not closed.

$p->get_token

This method will return the next token found in the HTML document, or undef at the end of the document. The token is returned as an array reference. The first element of the array will be a single character string denoting the type of this token; "S" for start tag, "E" for end tag, "T" for text, "C" for comment, and "D" for declaration. The rest of the array is the same as the arguments passed to the corresponding HTML::Parser callbacks (see HTML::Parser). This summarize the tokens that can occur:

  ["S", $tag, %$attr, @$attrseq, $origtext]
  ["E", $tag, $origtext]
  ["T", $text]
  ["C", $text]
  ["D", $text]

$p->unget_token($token,...)

If you find out you have read too many tokens you can push them back, so that they are returned the next time $p->get_token is called.

$p->get_tag( [$tag] )

This method return the next start or end tag (skipping any other tokens), or undef if there is no more tags in the document. If an argument is given, then we skip tokens until the specified tag is found. A tag is returned as an array reference of the same form as for $p->get_token above, but the type code (first element) is missing and the name of end tags are prefixed with "/". This means that the tags returned look like this:

  [$tag, %$attr, @$attrseq, $origtext]
  ["/$tag", $origtext]

$p->get_text( [$endtag] )

This method returns all text found at the current position. It will return a zero length string if the next token is not text. The optional $endtag argument specify that any text occurring before the given tag is to be returned. Any entities will be expanded to their corresponding character.

The $p->{textify} attribute is a hash that define how certain tags can be treated as text. If the name of a start tag match a key in this hash then this tag is converted to text. The hash value is used to specify which tag attribute to obtain the text from. If this tag attribute is missing, then the upper case name of the tag enclosed in brackets is returned, e.g. "[IMG]". The hash value can also be a subroutine reference. In this case the routine is called with the start tag token content as arguments and the return values is treated as the text.

The default $p->{textify} value is:

  {img => "alt", applet => "alt"}

This means that <IMG> and <APPLET> tags are treated as text, and that the text to substitute can be found as ALT attribute.

$p->get_trimmed_text( [$endtag] )

Same as $p->get_text above, but will collapse any sequence of white space to a single space character. Leading and trailing space is removed.

EXAMPLES

This example extract all links from a document. It will print one line for each link, containing the URL and the textual description between the <A>...</A> tags:

  use HTML::TokeParser;
  $p = HTML::TokeParser->new(shift||"index.html");

  while (my $token = $p->get_tag("a")) {
      my $url = $token->[1]{href} || "-";
      my $text = $p->get_trimmed_text("/a");
      print "$url\t$text\n";
  }

This example extract the <TITLE> from the document:

  use HTML::TokeParser;
  $p = HTML::TokeParser->new(shift||"index.html");
  if ($p->get_tag("title")) {
      my $title = $p->get_trimmed_text;
      print "Title: $title\n";
  }

SEE ALSO

HTML::Parser

COPYRIGHT

Copyright 1998-1999 Gisle Aas.

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.