Search::ESsearcher::Templates::httpAccess - Provicdes support for HTTP access logs sucked down via beats.
Version 0.0.0
This uses a logstath beasts input akin to below.
The important bit below is setting the "type" to "beats" and "fields.log" to "apache-access".
If you are using something different than "type" and "beats" you can specify that via "--field" and "--fieldv" respectively.
If you are using something different than "fields.log" and "apache-access" you can specify that via "--field2" and "--field2v" respectively.
input { beats { host => "192.168.14.3" port => 5044 type => "beats" } } filter { if [fields][log] == "apache-access" { grok { match => { "message" => "%{HTTPD_COMBINEDLOG}+%{GREEDYDATA:extra_fields}" } overwrite => [ "message" ] } mutate { convert => ["response", "integer"] convert => ["bytes", "integer"] convert => ["responsetime", "float"] } geoip { source => "clientip" target => "geoip" add_tag => [ "apache-geoip" ] } date { match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ] remove_field => [ "timestamp" ] } useragent { source => "agent" } } } output { if [type] == "beats" { elasticsearch { hosts => [ "127.0.0.1:9200" ] } } }
Then for file beats, something akin to below. The really important bits here the various values for "fields".
For "fields.vhost" and "fields.vhost_port" if you are using somethind different, you can specify that via "--field3" and "--field4" respectively.
- type: log enabled: true paths: - /var/log/apache/foo.bar:80-access.log fields: log: apache-access vhost: foo.bar vhost_port: 80
The machine beasts is running on feeding info to logstash/ES.
The response code from the HTTP server.
The verb used with the request.
The domain served up.
The port for the vhost.
The client IP that made the request.
The supplied OS value that made the request.
Shows the OS value.
The HTTP request.
The supplied referrer for the request.
The supplied agent value that made the request.
Do not show the agent field.
The authed user for the request.
Response bytes greater than.
Response bytes greater than or equal to.
Response bytes less than.
Response bytes less than or equal to.
Require GEO IP to have worked.
The 2 letter country code.
Show country code.
The state/province/etc to search for.
Show region code.
The postal code to search for.
Show postal code.
The city to search for.
Show city name.
The number of items to return.
Date greater than.
Date greater than or equal to.
Date less than.
Date less than or equal to.
Messages to match.
, OR + AND ! NOT
A list seperated by any of those will be transformed
These may be used with program, facility, pid, or host.
example: --program postfix,spamd results: postfix OR spamd
date
/^-/ appends "now" to it. So "-5m" becomes "now-5m".
/^u\:/ takes what is after ":" and uses Time::ParseDate to convert it to a unix time value.
Any thing not matching maching any of the above will just be passed on.
To install Search::ESsearcher, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Search::ESsearcher
CPAN shell
perl -MCPAN -e shell install Search::ESsearcher
For more information on module installation, please visit the detailed CPAN module installation guide.