HTML to XML converter for use with NexTrieve.
XML <filename> attribute contains the filename of the HTML file.
XML <title> attribute contains the title of the HTML-file (if any)
XML <title> text-type contains the title of the HTML-file (if any)
XML <description> text-type contains the text that was found in the CONTENT= attribute of the HTML <META> tag with the name "DESCRIPTION".
XML <keywords> text-type contains the text that was found in the CONTENT= attribute of the HTML <META> tag with the name "KEYWORDS".
Any text within an HTML <SCRIPT> container is ignored.
Any text in any other HTML container is added to the XML <text> container.
<document> <attributes> <filename>index.html</filename> <title>Index of contents</title> </attributes> <text> <title>Index of contents</title> <description>Description of file found in meta tags</description> <keywords>Keywords of file found in meta tags</keywords> Text found in the body of the file </text> </document>
A match for a word in a query in the <title> text-type can be very much more significant than when that word would be found in the body text. However, if you want to display the information of a hit, it is handy to have the title (and the filename) of the document available as well. That is why the title is added as an attribute as well, even though it can not be used for constraining a query (at least, not yet).
html2ntvml -f file1 file2 file3 > xml html2ntvml <files.list > xml
Convert all .html files in the "doc_root" directory to XML and store that XML in the file "xml".
html2ntvml -f doc_root/*.html >xml
Index all of the files located by find command.
find / --iregex '*.htm*' | html2ntvml | docseq | ntvindex -
Requires the availability of the NexTrieve.pm module and associated modules as found on CPAN (http://www.cpan.org/).
If you want to specify the filenames from the command line rather than pipe them from STDIN, that is possible by specifying the -f parameter, followed by the list of files you want to process. This is additional to any filenames piped through STDIN.
When processing HTML, many older HTML-files do not contain the information needed for XML to determine which character encoding is being used. By specifying the -E parameter, you can specify which encoding should be assumed if no encoding information is found. The default is "iso-8859-1".
If you are merging multiple runs of this script into the same file, you do not need the <?xml...?> processor instruction to be repeated. By specifying this flag, the processor instruction will _not_ be emitted to the XML stream.
If you are merging results of multiple runs, you may also want the <ntv:docseq> container not to be emitted. Specifying the -b (for "bare XML") flag does just that.
Some HTML out there in the world contains very long titles. This is done by some people to get higher rankings, as many search engines value text in a title more than text in a body (or only search in the title at all).
Experience has shown that titles of more than 10K are not uncommon. This however causes all sorts of problems in the display of the hitlists (where the title is one of the attributes returned) and in general it brings down the performance.
The -t parameter allows you to put a maximum length of the title as stored as an attribute (and therefore returned in the hitlist). It does not alter the length of the title stored as a texttype.
The default for -t is 0, indicating not limiting of text.
Sometimes a binary file (such as .gif image) can become part of the list of files to be processed, causing garbage to be indexed. If the -i flag is specified, an additional check is performed to guess whether the file is a binary file. If it is, then it will be silently ignored.
If the HTML-files processed are in fact ASP-pages, it may be wise to remove the ASP-style tags from the HTML before being processed. Specifying the "-a" flag will do just that: removing anything between "<%...%>" inclusive.
If the HTML-files processed are in fact PHP-pages, it may be wise to remove the PHP-style tags from the HTML being processed. Specifying the "-p" will do just that: removing anything between "<?...?>" and "<%..%>" inclusive. Please note that the <script language="php">...</script> tags are always removed, regardless of the -p setting.
Elizabeth Mattijsen, <liz@dijkmat.nl>.
Please report bugs to <perlbugs@dijkmat.nl>.
Copyright (c) 1995-2002 Elizabeth Mattijsen <liz@dijkmat.nl>. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
http://www.nextrieve.com and the NexTrieve::xxx modules.
To install NexTrieve, copy and paste the appropriate command in to your terminal.
cpanm
cpanm NexTrieve
CPAN shell
perl -MCPAN -e shell install NexTrieve
For more information on module installation, please visit the detailed CPAN module installation guide.