NAME

Search::ESsearcher::Templates::httpAccess - Provicdes support for HTTP access logs sucked down via beats.

VERSION

Version 0.0.0

LOGSTASH / FILEBEAT

This uses a logstath beasts input akin to below.

The important bit below is setting the "type" to "beats" and "fields.log" to "apache-access".

If you are using something different than "type" and "beats" you can specify that via "--field" and "--fieldv" respectively.

If you are using something different than "fields.log" and "apache-access" you can specify that via "--field2" and "--field2v" respectively.

    input {
      beats {
        host => "192.168.14.3"
        port => 5044
        type => "beats"
      }
    }
    
    filter {
        if [fields][log] == "apache-access" {
                    grok {
                            match => {
                                    "message" => "%{HTTPD_COMBINEDLOG}+%{GREEDYDATA:extra_fields}"
                            }
                            overwrite => [ "message" ]
                    }
    
                    mutate {
                            convert => ["response", "integer"]
                            convert => ["bytes", "integer"]
                            convert => ["responsetime", "float"]
                    }
                    geoip {
                            source => "clientip"
                            target => "geoip"
                            add_tag => [ "apache-geoip" ]
                    }
                    date {
                            match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
                            remove_field => [ "timestamp" ]
                    }
                    useragent {
                            source => "agent"
                    }
        }
    }
    
    output {
      if [type] == "beats" {
        elasticsearch {
          hosts => [ "127.0.0.1:9200" ]
        }
      }
    }

Then for file beats, something akin to below. The really important bits here the various values for "fields".

For "fields.vhost" and "fields.vhost_port" if you are using somethind different, you can specify that via "--field3" and "--field4" respectively.

    - type: log
      enabled: true
      paths:
        - /var/log/apache/foo.bar:80-access.log
      fields:
         log: apache-access
         vhost: foo.bar
         vhost_port: 80

Options

--host <host>

The machine beasts is running on feeding info to logstash/ES.

--response <code>

The response code from the HTTP server.

--verb <verb>

The verb used with the request.

--vhost <vhost>

The domain served up.

--port <port>

The port for the vhost.

--ip <ip>

The client IP that made the request.

--os <os>

The supplied OS value that made the request.

--showos

Shows the OS value.

--req <req>

The HTTP request.

--ref <ref>

The supplied referrer for the request.

--agent <agent>

The supplied agent value that made the request.

--noagent

Do not show the agent field.

--auth <auth>

The authed user for the request.

--bgt <bytes>

Response bytes greater than.

--bgte <bytes>

Response bytes greater than or equal to.

--blt <bytes>

Response bytes less than.

--blte <bytes>

Response bytes less than or equal to.

--geoip

Require GEO IP to have worked.

--country <country>

The 2 letter country code.

--showcountry

Show country code.

--region <state>

The state/province/etc to search for.

--showregion

Show region code.

--postal <zipcode>

The postal code to search for.

--showpostal

Show postal code.

--city <cide>

The city to search for.

--showcity

Show city name.

--size <count>

The number of items to return.

--dgt <date>

Date greater than.

--dgte <date>

Date greater than or equal to.

--dlt <date>

Date less than.

--dlte <date>

Date less than or equal to.

--msg <message>

Messages to match.

AND, OR, or NOT shortcut

    , OR
    + AND
    ! NOT

A list seperated by any of those will be transformed

These may be used with program, facility, pid, or host.

    example: --program postfix,spamd
    
    results: postfix OR spamd

date

date

/^-/ appends "now" to it. So "-5m" becomes "now-5m".

/^u\:/ takes what is after ":" and uses Time::ParseDate to convert it to a unix time value.

Any thing not matching maching any of the above will just be passed on.

cpanm Search::ESsearcher

perl -MCPAN -e shell
install Search::ESsearcher