NAME
files-to-elasticsearch.pl - A simple utility to tail a file and index each line as a document in ElasticSearch
VERSION
version 0.002
SYNOPSIS
To see available options, run:
file-to-elasticsearch.pl --help
Create a config file and run the utility:
file-to-elasticsearch.pl --config config.yaml --log4perl logging.conf --debug
This will run a single threaded POE instance that will tail the log files you've requested, performing the requested transformations and sending them to the elasticsearch cluster and index you've specified.
CONFIGURATION
The configuration file is YAML and looks like:
---
elasticsearch:
servers: [ "localhost:9200" ]
type: "log"
index: "syslog-%Y.%m.%d"
tail:
- file: '/var/log/osquery/result.log'
index: "osquery-result-%Y.%m.%d"
decode: json
extract:
- by: split
from: name
when: '^pack'
into: 'pack'
split_on: '/'
split_parts: [ null, "name", "report" ]
mutate:
prune: true
remove: [ "calendarTime", "epoch", "counter", "_raw" ]
rename:
unixTime: _epoch
AUTHOR
Brad Lhotsky <brad@divisionbyzero.net>
COPYRIGHT AND LICENSE
This software is Copyright (c) 2018 by Brad Lhotsky.
This is free software, licensed under:
The (three-clause) BSD License