The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.
 / 
App-JsonLogUtils-0.03
River stage zero No dependents
/ App::JsonLogUtils

NAME

App::JsonLogUtils - Command line utilities for dealing with JSON-formatted log files

VERSION

version 0.03

SYNOPSIS

# From the command line
tail -f /path/to/log/file.log \
  | jgrep -m message="some pattern" \
  | jcut -f "timestamp priority message" \
  | cols -c "timestamp priority message" -s '|' \
  | column -t -s '|'


# From code
use App::JsonLogUtils qw(tail json_log);

my $log = json_log tail '/path/to/file.log';

while (my $entry = <$log>) {
  my ($json, $line) = @$entry;
  ...
}


# Grepping JSON logs
use App::JsonLogUtils qw(lines json_log);
use Iterator::Simple qw(igrep imap);

my $entries = igrep{ $_->{foo} =~ /bar/ } # filter objects
              imap{ $_->[0] }             # select the object
              json_log                    # parse
              lines '/path/to/file.log';  # read

DESCRIPTION

Writing logs in JSON, one object per line, makes them very easily machine readable. Wonderful. Unfortunately, it also makes it unfuriating to deal with them using the standard unix command line tools. This package provides a few tools to salve the burn.

COMMAND LINE TOOLS

jgrep

Greps patterns in individual object fields.

jcut

Filter the fields included in objects.

jcols

Display fields in a format suitable for column.

jshell

An interactive shell for monitoring JSON log files.

EXPORTABLE ROUTINES

If desired, the iterators used to implement the tools above are optionally exported by the main module.

lines

Accepts a file path or opened file handle and returns an iterator which yields the chomped lines from the file.

my $log = lines '/path/to/file.log';

while (my $line = <$log>) {
  ...
}

tail

Accepts a file path or opened file handle and returns an iterator while yields chomped lines from the file as they are appended, starting from the end of the file. Lines already written to the file when this routine is first called are ignored (that is, there is no equivalent to tail -c 10 at this time).

my $log = tail '/path/to/file.log';

while (my $line = <$log>) { # sleeps until lines appended to file
  ...
}

json_log

Accepts a file iterator (see "tail" and "lines") and returns an iterator yielding an array ref holding two items, a hash ref of the parsed JSON log entry, and the original log entry string. Empty lines are skipped with a warning. JSON decoding errors are ignored with a warning.

my $lines = json_log tail '/path/to/file.log';

while (my $entry = <$lines>) {
  my ($object, $line) = @_;
  ...
}

json_cols

Accepts a list of fields (as a space-separared string or array ref of strings), a string separator, and an iterator over JSON object strings, and returns a new iterator. The returned iterator will first yield a string of column names joined by the separator string. Subsequent calls will iterate over the JSON object strings, return the value of each of the selected fields joined by the separator string.

# File $input
{"a": 1, "b": 2, "c": 3}
{"a": 4, "b": 5, "c": 6}
{"a": 7, "b": 8, "c": 9}

# Select columns a and c, separated by a pipe
my $cols = json_cols "a c", "|" , lines $input;

# ...yields the following strings:
"a|c"
"1|3"
"4|6"
"7|9"

json_cut

Accepts a space-separated string or array ref of $fields, boolean $inverse, and an iterator of JSON log lines. Returns an iterator yielding objects containing only the fields selected in $fields. If $inverse is true, instead yields objects containing only the fields not contained in $fields.

Give the same input as the previous example:

my $cut = json_cut "a c", 0, lines $input;

# ...yields the following hash refs:
{a => 1, c => 3}
{a => 4, c => 6}
{a => 7, c => 9}

# Inverted
my $cut = json_cut "a c", 1, lines $input;

# ...yields:
{b => 2}
{b => 5}
{b => 8}

json_grep

Accepts a hash ref where keys are field names and values are arrays of regular expressions, a boolean $inverse, and an iterator of JSON object strings. Returns an iterator yielding array refs of the parsed JSON object hash and the original string (just like "json_log"). Only those entries for which all fields' patterns match are returned. If $inverse is set, the logic is negated and only those entries for which all patterns test false are returned.

# File $input
{"foo": "bar"}
{"foo": "baz"}
{"foo": "bat"}
{"foo": "BAR"}

# Code
my $entries = json_grep { foo => [qw/bar/i, qr/baz/] }, 0, lines $input;

# ...yields the following:
[ {foo => "bar"}, '{"foo": "bar"}' ]
[ {foo => "baz"}, '{"foo": "baz"}' ]
[ {foo => "BAR"}, '{"foo": "BAR"}' ]

FUTURE PLANS

None, but will happily consider requests and patches.

AUTHOR

Jeff Ober <sysread@fastmail.fm>

COPYRIGHT AND LICENSE

This software is copyright (c) 2018 by Jeff Ober.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.