NAME

Go::Tokenize - Tokenize Go

SYNOPSIS

    use Go::Tokenize;

VERSION

This documents version 0.02 of Go-Tokenize corresponding to git commit b750cdd6ab0a0d951dd19c1c8bf37c54f8ea73d5 released on Sat Mar 13 17:18:20 2021 +0900.

DESCRIPTION

Turn Go code into tokens.

FUNCTIONS

tokenize

    my $tokens = tokenize ($go);

Given your Go program text in $go, turn it into tokens. The return value is an array reference which contains hash references. The hash references have the following keys:

contents

The text content of the token.

end

The position of the final character of the token, in characters. See "start" for the numbering. This is the value of pos as the regex used for matching matches components.

line

The line number of the start position of the token, numbered from 1.

start

The position of the beginning character of the token, in characters. This is numbered from 0 for the first character of the file. If you read the Go file in using :encoding(utf8) and the file contains non-ASCII characters, this and "end" will be different from the byte offsets. Also, because this is the position of the character, if the token is one character in length, its value will be the same as "end". In other words the length of the token is (end - start + 1).

type

The type of the token. Currently the following are supported:

comment

identifier

integer

keyword

numeric

operator

rune

string

whitespace

DEPENDENCIES

C::Tokenize

This is used because Go comments are identical to C comments.

Text::LineNumber

AUTHOR

Ben Bullock, <bkb@cpan.org>

COPYRIGHT & LICENCE

This package and associated files are copyright (C) 2021 Ben Bullock.

You can use, copy, modify and redistribute this package and associated files under the Perl Artistic Licence or the GNU General Public Licence.