App::ElasticSearch::Utilities::QueryString - CLI query string fixer
version 8.8
This class provides a pluggable architecture to expand query strings on the command-line into complex Elasticsearch queries.
Defaults to 'query', but can also be set to 'filter' so the elements will be added to the 'must' or 'filter' parameter.
An array reference of additional namespaces to search for loading the query string processing plugins. Example:
$qs->search_path([qw(My::Company::QueryString)]);
This will search:
App::ElasticSearch::Utilities::QueryString::* My::Company::QueryString::*
For query processing plugins.
When fixing up the query string, if two tokens are found next to eachother missing a joining token, join using this token. Can be either AND or OR, and defaults to AND.
AND
OR
Array reference of ordered query string processing plugins, lazily assembled.
A hash reference with the field data from App::ElasticSearch::Utilities::es_index_fields.
This function takes a list of tokens, often from the command line via @ARGV. Uses a plugin infrastructure to allow customization.
Returns: App::ElasticSearch::Utilities::Query object
The token expansion plugins can return undefined, which is basically a noop on the token. The plugin can return a hash reference, which marks that token as handled and no other plugins receive that token. The hash reference may contain:
This is the rewritten bits that will be reassembled in to the final query string.
This is usually a hash reference representing the condition going into the bool query. For instance:
{ terms => { field => [qw(alice bob charlie)] } }
Or
{ prefix => { user_agent => 'Go ' } }
These conditions will wind up in the must or must_not section of the bool query depending on the state of the the invert flag.
This is used by the bareword "not" to track whether the token invoked a flip from the must to the must_not state. After each token is processed, if it didn't set this flag, the flag is reset.
This is used for bare words like "not", "or", and "and" to denote that these terms cannot dangle from the beginning or end of the query_string. This allows the final pass of the query_string builder to strip these words to prevent syntax errors.
The search string is pre-analyzed before being sent to ElasticSearch. The following plugins work to manipulate the query string and provide richer, more complete syntax for CLI applications.
The following barewords are transformed:
or => OR and => AND not => NOT
Provides field prefixes to manipulate the text search capabilities.
Provide an '=' prefix to a query string parameter to promote that parameter to a term filter.
term
This allows for exact matches of a field without worrying about escaping Lucene special character filters.
E.g.:
user_agent:"Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1"
Is evaluated into a weird query that doesn't do what you want. However:
=user_agent:"Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1"
Is translated into:
{ term => { user_agent => "Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1" } }
Provide an '*' prefix to a query string parameter to promote that parameter to a wildcard filter.
wildcard
This uses the wild card match for text fields to making matching more intuitive.
*user_agent:"Mozilla*"
{ wildcard => { user_agent => "Mozilla* } }
Provide an '/' prefix to a query string parameter to promote that parameter to a regexp filter.
regexp
If you want to use regexp matching for finding data, you can use:
/message:'\\bden(ial|ied|y)'
{ regexp => { message => "\\bden(ial|ied|y)" } }
Provide an '~' prefix to a query string parameter to promote that parameter to a fuzzy filter.
fuzzy
~message:deny
{ fuzzy => { message => "deny" } }
Provide an '+' prefix to a query string parameter to promote that parameter to a match_phrase filter.
match_phrase
+message:"login denied"
{ match_phrase => { message => "login denied" } }
If the field meta data is provided and the field is a text type, the query will automatically be mapped to a match query.
text
match
# message field is text message:"foo"
{ match => { message => "foo" } }
If a field is an IP address uses CIDR Notation, it's expanded to a range query.
src_ip:10.0/8 => src_ip:[10.0.0.0 TO 10.255.255.255]
This plugin translates some special comparison operators so you don't need to remember them anymore.
Example:
price:<100
Will translate into a:
{ range: { price: { lt: 100 } } }
And:
price:>50,<100
Will translate to:
{ range: { price: { gt: 50, lt: 100 } } }
gt via >, gte via >=, lt via <, lte via <=
This plugin translates some special underscore surrounded tokens into the Elasticsearch Query DSL.
Implemented:
Example query string:
_prefix_:useragent:'Go '
Translates into:
{ prefix => { useragent => 'Go ' } }
If the match ends in .dat, .txt, .csv, or .json then we attempt to read a file with that name and OR the condition:
$ cat test.dat 50 1.2.3.4 40 1.2.3.5 30 1.2.3.6 20 1.2.3.7
$ cat test.csv 50,1.2.3.4 40,1.2.3.5 30,1.2.3.6 20,1.2.3.7
$ cat test.txt 1.2.3.4 1.2.3.5 1.2.3.6 1.2.3.7
$ cat test.json { "ip": "1.2.3.4" } { "ip": "1.2.3.5" } { "ip": "1.2.3.6" } { "ip": "1.2.3.7" }
We can source that file:
src_ip:test.dat => src_ip:(1.2.3.4 1.2.3.5 1.2.3.6 1.2.3.7) src_ip:test.json[ip] => src_ip:(1.2.3.4 1.2.3.5 1.2.3.6 1.2.3.7)
This make it simple to use the --data-file output options and build queries based off previous queries. For .txt and .dat file, the delimiter for columns in the file must be either a tab or a null. For files ending in .csv, Text::CSV_XS is used to accurate parsing of the file format. Files ending in .json are considered to be newline-delimited JSON.
You can also specify the column of the data file to use, the default being the last column or (-1). Columns are zero-based indexing. This means the first column is index 0, second is 1, .. The previous example can be rewritten as:
src_ip:test.dat[1]
or: src_ip:test.dat[-1]
For newline delimited JSON files, you need to specify the key path you want to extract from the file. If we have a JSON source file with:
{ "first": { "second": { "third": [ "bob", "alice" ] } } } { "first": { "second": { "third": "ginger" } } } { "first": { "second": { "nope": "fred" } } }
We could search using:
actor:test.json[first.second.third]
Which would expand to:
{ "terms": { "actor": [ "alice", "bob", "ginger" ] } }
This option will iterate through the whole file and unique the elements of the list. They will then be transformed into an appropriate terms query.
We can also have a group of wildcard or regexp in a file:
$ cat wildcards.dat *@gmail.com *@yahoo.com
To enable wildcard parsing, prefix the filename with a *.
*
es-search.pl to_address:*wildcards.dat
Which expands the query to:
{ "bool": { "minimum_should_match":1, "should": [ {"wildcard":{"to_outbound":{"value":"*@gmail.com"}}}, {"wildcard":{"to_outbound":{"value":"*@yahoo.com"}}} ] } }
No attempt is made to verify or validate the wildcard patterns.
If you'd like to specify a file full of regexp, you can do that as well:
$ cat regexp.dat .*google\.com$ .*yahoo\.com$
To enable regexp parsing, prefix the filename with a ~.
~
es-search.pl to_address:~regexp.dat
{ "bool": { "minimum_should_match":1, "should": [ {"regexp":{"to_outbound":{"value":".*google\\.com$"}}}, {"regexp":{"to_outbound":{"value":".*yahoo\\.com$"}}} ] } }
No attempt is made to verify or validate the regexp expressions.
Implement the proposed nested query syntax early. Example:
nested_path:"field:match AND string"
Brad Lhotsky <brad@divisionbyzero.net>
This software is Copyright (c) 2024 by Brad Lhotsky.
This is free software, licensed under:
The (three-clause) BSD License
To install App::ElasticSearch::Utilities, copy and paste the appropriate command in to your terminal.
cpanm
cpanm App::ElasticSearch::Utilities
CPAN shell
perl -MCPAN -e shell install App::ElasticSearch::Utilities
For more information on module installation, please visit the detailed CPAN module installation guide.