NAME

log-defer-viz - command-line utility for rendering log messages created by Log::Defer

DESCRIPTION

Log::Defer is a module that creates structured logs. Read its documentation which explains how structured logging can help you.

This module installs a command-line script that parses these logs and displays them in a readable manner.

INPUT METHODS

    $ cat file.log | log-defer-viz
    $ log-defer-viz < file.log
    $ log-defer-viz file.log
    $ log-defer-viz file.log file2.log
    $ log-defer-viz archived.log.gz more_logs.bz2

INPUT FORMAT

    $ log-defer-viz --input-format=json  ## default is newline separated JSON
    $ log-defer-viz --input-format=sereal  ## Sereal::Decoder (not impl)
    $ log-defer-viz --input-format=messagepack  ## Data::MessagePack (not impl)
    $ log-defer-viz --input-format=storable  ## Storable (not impl)

Note: The only input format currently implemented is newline-separated JSON.

LOG MESSAGES

    $ log-defer-viz  ## by default shows error, warn, and info logs
    $ log-defer-viz -v  ## verbose mode (adds debug logs and more)
    $ log-defer-viz --debug  ## show debug logs
    $ log-defer-viz --quiet  ## only errors and warnings
    $ log-defer-viz --verbosity 25  ## numeric verbosity threshold
    $ log-defer-viz --nowarn  ## muffle warn logs (so show error and info)
    $ log-defer-viz --nologs  ## don't show log section
    $ log-defer-viz --nocolour  ## turn off terminal colours

TIMERS

    $ log-defer-viz --timer-columns 80  ## width of timer chart
    $ log-defer-viz --since-now  ## show relative to now times
                                 ##   like "34 minutes ago"
    $ log-defer-viz --notimers  ## don't show timer chart

DATA SECTION

Data is extra embedded information in the log file. The available outputs are pretty-json, json, yaml, and dumper.

    $ log-defer-viz --data  ## show data section. default is pretty-json
    $ log-defer-viz --data-format=json  ## compact, not pretty
    $ log-defer-viz --data-format=dumper  ## Data::Dumper
    $ log-defer-viz --data-only  ## only show data

COUNT

The count parameter provides a method to count the number of times a specific key/value appears in the data section of the log file.

    $ log-defer-viz --data --count ip_address ## Display how many log lines for each ip address
    $ log-defer-viz --data --count ip_address --count user_id ## Display how many log lines for each ip address and each user_id

MISC

    $ log-defer-viz --help  ## the text you're reading now
    $ log-defer-viz --grep '$_->{data}'  ## grep for records that have a data section.
                                         ## $_ is the entire Log::Defer entry.
    $ log-defer-viz --map '$_->{data}->{username}'  ## Extract username from data

GREPING

As shown above, there is a --grep command-line option. This lets you filter log messages using arbitrary perl code. If the expression returns true, the log message is processed and displayed as normal.

Being able to do this easily is an important advantage of structured logs. With unstructured logs it is often difficult to extract all of the information related to a request and nothing else.

For example, here is how to grep for all requests that took longer than 500 milliseconds:

    $ log-defer-viz --grep '$_->{end} > .5' server.log

Depending on your underlying storage format, it may be meaningful to grep before passing to log-defer-viz. Currently the only supported storage format is newline-separated JSON which is designed to be pre-grepable. If your search string appears anywhere in the object, the entire log message will be displayed:

    $ grep 10.9.1.2 app.log | log-defer-viz

The final and most error-prone way to grep Log::Defer logs is to grep the unstructured output of log-defer-viz:

    $ log-defer-viz app.log | grep 10.9.1.2

MAPPING

Similar to --grep, there is also a --map command-line option. If this option is passed in, the only thing that is output is whatever your --map expression returns.

For example, if you are putting the PID into the data section with $log->data->{pid} = $$, then you can extract the PID like so:

    $ log-defer-viz --map '$_->{data}->{pid}' app.log
    9765
    9768
    9771

As with --grep, you have access to any perl functions you need including JSON::XS. Also, you can combine --map and --grep. For example, here is how to do a "pass-through" grep where the output is another valid JSON-encoded Log::Defer file:

    $ log-defer-viz -g '_->{data}->{username} eq "jimmy"' \
                    -m 'encode_json _'                    \
                    app.log                               \
                    > jimmys-requests.log

Note that the above also takes advantage of the -g and -m option shortcuts for --map and --grep respectively. It also uses the special _ shortcut which is an abbreviation for $_.

SEE ALSO

Log::Defer

Log::Defer::Viz github repo

AUTHOR

Doug Hoyte, <doug@hcsw.org>

COPYRIGHT & LICENSE

Copyright 2013 Doug Hoyte.

This module is licensed under the same terms as perl itself.

cpanm Log::Defer::Viz

perl -MCPAN -e shell
install Log::Defer::Viz