log-defer-viz - command-line utility for rendering log messages created by Log::Defer
Log::Defer is a module that creates structured logs. The Log::Defer documentation explains structured logging and its benefits over ad-hoc logging.
This module installs a command-line script that parses structured logs created by Log::Defer and displays them in a readable manner.
$ cat file.log | log-defer-viz $ log-defer-viz < file.log $ log-defer-viz file.log $ log-defer-viz -f file.log # continuously tail file $ log-defer-viz file.log file2.log $ log-defer-viz archived.log.gz more_logs.bz2
$ log-defer-viz --input-format=json ## default is newline separated JSON $ log-defer-viz --input-format=sereal ## Sereal::Decoder (not impl) $ log-defer-viz --input-format=messagepack ## Data::MessagePack (not impl) $ log-defer-viz --input-format=storable ## Storable (not impl)
Note: The only input format currently implemented is newline-separated JSON.
$ log-defer-viz ## by default shows error, warn, and info logs $ log-defer-viz -v ## verbose mode (adds debug logs and more) $ log-defer-viz --debug ## show debug logs $ log-defer-viz --quiet ## only errors and warnings $ log-defer-viz --verbosity 25 ## numeric verbosity threshold $ log-defer-viz --nowarn ## muffle warn logs (so show error and info) $ log-defer-viz --nologs ## don't show log section $ log-defer-viz --nocolour ## turn off terminal colours $ log-defer-viz --preserve-newlines # dont indent log messages to align with log-defer-viz output
$ log-defer-viz --timer-columns 80 ## width of timer chart $ log-defer-viz --since-now ## show relative to now times ## like "34 minutes ago" $ log-defer-viz --notimers ## don't show timer chart $ log-defer-viz --tz UTC ## show times in UTC, not local
Applications can optionally log information in a "data" hash. This information is mostly designed to be extracted by programs so log-defer-viz doesn't display it by default. Use the --data option to display it anyway, and the --data-format option to choose the format to display it in. The available formats are pretty-json, json, yaml, and dumper.
log-defer-viz
--data
--data-format
pretty-json
json
yaml
dumper
$ log-defer-viz --data ## show data section. default is pretty-json $ log-defer-viz --data-format=json ## compact, not pretty $ log-defer-viz --data-format=dumper ## Data::Dumper $ log-defer-viz --data-only ## only show data
The count parameter provides a method to count the number of times a specific key/value appears in the data section of the log file.
$ log-defer-viz --data --count ip_address ## Display how many log lines for each ip address $ log-defer-viz --data --count ip_address --count user_id ## Display how many log lines for each ip address and each user_id
$ log-defer-viz --help ## the text you're reading now $ log-defer-viz --grep '$_->{data}' ## grep for records that have a data section. ## $_ is the entire Log::Defer entry. $ log-defer-viz --map '$_->{data}->{username}' ## Extract username from data
As shown above, there is a --grep command-line option. This lets you filter log messages using arbitrary perl code. If the expression returns true, the log message is processed and displayed as usual.
--grep
Being able to do this easily is an important advantage of structured logs. With unstructured logs it is often difficult to extract all of the information related to a request and nothing else.
For example, here is how to grep for all requests that took longer than 500 milliseconds:
$ log-defer-viz --grep '$_->{end} > .5' server.log
Depending on your underlying storage format, it may be meaningful to grep before passing to log-defer-viz. Currently the only supported storage format is newline-separated JSON which is designed to be pre-grepable. If your search string appears anywhere in the object, the entire log message will be displayed:
$ grep 10.9.1.2 app.log | log-defer-viz
The final and most error-prone way to grep Log::Defer logs is to grep the unstructured output of log-defer-viz (not recommended):
$ log-defer-viz app.log | grep 10.9.1.2
Similar to --grep, there is also a --map command-line option. If this option is passed in, the only output is whatever your --map expression returns.
--map
For example, if you are putting the PID into the data section with $log->data->{pid} = $$, then you can extract the PID like so:
$log->data->{pid} = $$
$ log-defer-viz --map '$_->{data}->{pid}' < app.log 9765 9768 9771
Join together fields with a pipe:
$ log-defer-viz --map 'join "|", $_->{data}{pid}, $_->{start}' < app.log 9765|1362166673.95104 9768|1362168038.85611 9771|1362169482.39561
Make dates readable (localtime in scalar context makes a timestamp readable):
localtime
$ log-defer-viz --map 'join "|", $_->{data}{pid}, "".localtime($_->{start})' < app.log 9765|Fri Mar 1 14:37:53 2013 9768|Fri Mar 1 15:00:38 2013 9771|Fri Mar 1 15:24:42 2013
As with --grep, you have access to any perl functions you might need. Also, you can combine --map and --grep. The grep filtering will be applied before the mapping.
For example, here is how to do a "pass-through grep" where the output is another valid JSON-encoded Log::Defer file:
$ export USER=jimmy $ log-defer-viz -g "_->{data}{username} eq '$USER'" \ -m "encode_json _" \ < requests.log \ > jimmys-requests.log
Note that the above also demonstrates two shortcut features: First, the -g and -m switches are abbreviations for --grep and --map respectively. Second, in grep and map expressions the _ function is an abbreviation for $_. Although this is one character shorter, the main reason this exists is so that you can use double-quoted strings without having to worry about escaping $ characters from your shell.
-g
-m
_
$_
$
Log::Defer
Log::Defer::Viz github repo
The log-defer-viz is also useful for visualising logs created by Michael Pucyk's LogDefer Python module (since it outputs the same format)
Doug Hoyte, <doug@hcsw.org>
<doug@hcsw.org>
Matt Phillips, <mattp@cpan.org>
<mattp@cpan.org>
Copyright 2013-2014 Doug Hoyte and contributors.
This module is licensed under the same terms as perl itself.
To install Log::Defer::Viz, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Log::Defer::Viz
CPAN shell
perl -MCPAN -e shell install Log::Defer::Viz
For more information on module installation, please visit the detailed CPAN module installation guide.