++ed by:

2 PAUSE users
1 non-PAUSE user.

Doug Hoyte


log-defer-viz - command-line utility for rendering log messages created by Log::Defer


Log::Defer is a module that creates structured logs. The Log::Defer documentation explains structured logging and its benefits over ad-hoc logging.

This module installs a command-line script that parses structured logs created by Log::Defer and displays them in a readable manner.


    $ cat file.log | log-defer-viz
    $ log-defer-viz < file.log
    $ log-defer-viz file.log
    $ log-defer-viz -f file.log # continuously tail file
    $ log-defer-viz file.log file2.log
    $ log-defer-viz archived.log.gz more_logs.bz2


    $ log-defer-viz --input-format=json  ## default is newline separated JSON
    $ log-defer-viz --input-format=sereal  ## Sereal::Decoder (not impl)
    $ log-defer-viz --input-format=messagepack  ## Data::MessagePack (not impl)
    $ log-defer-viz --input-format=storable  ## Storable (not impl)

Note: The only input format currently implemented is newline-separated JSON.


    $ log-defer-viz  ## by default shows error, warn, and info logs
    $ log-defer-viz -v  ## verbose mode (adds debug logs and more)
    $ log-defer-viz --debug  ## show debug logs
    $ log-defer-viz --quiet  ## only errors and warnings
    $ log-defer-viz --verbosity 25  ## numeric verbosity threshold
    $ log-defer-viz --nowarn  ## muffle warn logs (so show error and info)
    $ log-defer-viz --nologs  ## don't show log section
    $ log-defer-viz --nocolour  ## turn off terminal colours
    $ log-defer-viz --preserve-newlines # dont indent log messages to align with log-defer-viz output


    $ log-defer-viz --timer-columns 80  ## width of timer chart
    $ log-defer-viz --since-now  ## show relative to now times
                                 ##   like "34 minutes ago"
    $ log-defer-viz --notimers  ## don't show timer chart
    $ log-defer-viz --tz UTC  ## show times in UTC, not local


Applications can optionally log information in a "data" hash. This information is mostly designed to be extracted by programs so log-defer-viz doesn't display it by default. Use the --data option to display it anyway, and the --data-format option to choose the format to display it in. The available formats are pretty-json, json, yaml, and dumper.

    $ log-defer-viz --data  ## show data section. default is pretty-json
    $ log-defer-viz --data-format=json  ## compact, not pretty
    $ log-defer-viz --data-format=dumper  ## Data::Dumper
    $ log-defer-viz --data-only  ## only show data


The count parameter provides a method to count the number of times a specific key/value appears in the data section of the log file.

    $ log-defer-viz --data --count ip_address ## Display how many log lines for each ip address
    $ log-defer-viz --data --count ip_address --count user_id ## Display how many log lines for each ip address and each user_id


    $ log-defer-viz --help  ## the text you're reading now
    $ log-defer-viz --grep '$_->{data}'  ## grep for records that have a data section.
                                         ## $_ is the entire Log::Defer entry.
    $ log-defer-viz --map '$_->{data}->{username}'  ## Extract username from data


As shown above, there is a --grep command-line option. This lets you filter log messages using arbitrary perl code. If the expression returns true, the log message is processed and displayed as usual.

Being able to do this easily is an important advantage of structured logs. With unstructured logs it is often difficult to extract all of the information related to a request and nothing else.

For example, here is how to grep for all requests that took longer than 500 milliseconds:

    $ log-defer-viz --grep '$_->{end} > .5' server.log

Depending on your underlying storage format, it may be meaningful to grep before passing to log-defer-viz. Currently the only supported storage format is newline-separated JSON which is designed to be pre-grepable. If your search string appears anywhere in the object, the entire log message will be displayed:

    $ grep app.log | log-defer-viz

The final and most error-prone way to grep Log::Defer logs is to grep the unstructured output of log-defer-viz (not recommended):

    $ log-defer-viz app.log | grep


Similar to --grep, there is also a --map command-line option. If this option is passed in, the only output is whatever your --map expression returns.

For example, if you are putting the PID into the data section with $log->data->{pid} = $$, then you can extract the PID like so:

    $ log-defer-viz --map '$_->{data}->{pid}' < app.log

Join together fields with a pipe:

    $ log-defer-viz --map 'join "|", $_->{data}{pid}, $_->{start}' < app.log

Make dates readable (localtime in scalar context makes a timestamp readable):

    $ log-defer-viz --map 'join "|", $_->{data}{pid}, "".localtime($_->{start})' < app.log
    9765|Fri Mar  1 14:37:53 2013
    9768|Fri Mar  1 15:00:38 2013
    9771|Fri Mar  1 15:24:42 2013

As with --grep, you have access to any perl functions you might need. Also, you can combine --map and --grep. The grep filtering will be applied before the mapping.

For example, here is how to do a "pass-through grep" where the output is another valid JSON-encoded Log::Defer file:

    $ export USER=jimmy
    $ log-defer-viz -g "_->{data}{username} eq '$USER'" \
                    -m "encode_json _"                  \
                    < requests.log                      \
                    > jimmys-requests.log

Note that the above also demonstrates two shortcut features: First, the -g and -m switches are abbreviations for --grep and --map respectively. Second, in grep and map expressions the _ function is an abbreviation for $_. Although this is one character shorter, the main reason this exists is so that you can use double-quoted strings without having to worry about escaping $ characters from your shell.



Log::Defer::Viz github repo

The log-defer-viz is also useful for visualising logs created by Michael Pucyk's LogDefer Python module (since it outputs the same format)


Doug Hoyte, <doug@hcsw.org>


Matt Phillips, <mattp@cpan.org>


Copyright 2013-2014 Doug Hoyte and contributors.

This module is licensed under the same terms as perl itself.