RTF::Tokenizer - Tokenize RTF
Tokenizes RTF
use RTF::Tokenizer; # Create a tokenizer object my $tokenizer = RTF::Tokenizer->new(); # Populate it from a file $tokenizer->read_file('filename.txt'); # Or a file handle $tokenizer->read_file( \*STDIN ); # Or a string $tokenizer->read_string( '{\*\some rtf}' ); # Get the first token my ( $token_type, $argument, $parameter ) = $tokenizer->get_token();
This documentation assumes some basic knowledge of RTF. If you lack that, go read The_RTF_Cookbook:
http://search.cpan.org/search?dist=RTF-Writer
Returns a Tokenizer object. Currently accepts no arguments.
Fills the Tokenizer's buffer with the string, ready to start tokenizing.
Puts the first line from the filehandle into the buffer, and remembers the filehandle, so if you ask for a token, and the buffer is empty, it'll try and read the next line from the file.
Returns the next token as a 3 item list: 'type', 'argument', 'parameter'. Token is one of: text, control, group, or eof.
text
'type' is set to 'text'. 'argument' is set to the text itself. 'parameter' is left blank. NOTE: \{, \}, and \\ are all returned as control words, rather than rendered as text for you. As with \_, \- and friends.
control
'type' is 'control'. 'argument' is the control word or control symbol. 'parameter' is the control word's parameter if it has one - this will be numeric, EXCEPT when 'argument' is a literal ', in which case it will be a two-letter hex string.
group
'type' is 'group'. If it's the beginning of an RTF group, then 'argument' is 1, else if it's the end, argument is 0. 'parameter' is not set.
eof
End of file reached. 'type' is 'eof'. 'argument' is 1. 'parameter' is 0.
To avoid intrusively deep parsing, if an alternative ASCII representation is available for a unicode entity, and that ASCII representation contains {, or \, by themselves, things will go b<funky>. But I'm not convinced either of those are allowed by the spec.
Peter Sergeant <rtfr@clueball.com>
Copyright 2003 Peter Sergeant.
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
To install RTF::Tokenizer, copy and paste the appropriate command in to your terminal.
cpanm
cpanm RTF::Tokenizer
CPAN shell
perl -MCPAN -e shell install RTF::Tokenizer
For more information on module installation, please visit the detailed CPAN module installation guide.