==================================================
Changes from 2016-07-21 00:00:00 +0000 to present.
==================================================
----------------------------------------
version 5.9 at 2018-07-21 01:17:48 +0000
----------------------------------------
Change: 9b00c64735a21b74223fddcde44fff74772efd17
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-20 18:10:38 +0000
Bumping Release to 5.9
Change: 415c84da87571aa488a0f246d423e24fbf446e00
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-18 17:58:55 +0000
Fix another uninitialized error in the code
----------------------------------------
version 5.8 at 2018-07-16 19:14:45 +0000
----------------------------------------
Change: 3f276cdc5e5b8a1fc6215275b8b3c24e3d8ef066
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-16 12:14:45 +0000
Docs update for 5.8 release
Change: 12336ccb1ee03eaa294af46a1340af3cbb47f4c1
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-16 11:45:58 +0000
Fix a warning and improve output
* Fix a "use of uninitialized" warning for when an index is missing a
date string but matches the base pattern * Fix logic bug that meant
'--delete --delete-days 1' was a noop because
of using '<' instead of '<='
Change: e3b0899af37835e7e9d372788c31339cc3cf8f01
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-16 11:43:38 +0000
Bubble up errors so we know why we failed
Change: 9a07d69d5a7571d58951682c3d190083cf0a84df
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-16 11:25:16 +0000
Show shards breakdown by primary and replica
----------------------------------------
version 5.7 at 2018-07-06 05:34:05 +0000
----------------------------------------
Change: e90729b13bb4fc45d68e9711ac139c8f8ab1dad8
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-05 22:34:05 +0000
Documentation Updates for 5.7 release
Change: 7498a4ccea72cd4cc8860316bf3aa175e278c158
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-05 22:23:55 +0000
Add support for statistical aggregations via --with
Change: 8a86345e449b355257358de7f81414d278442a38
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-05 22:05:43 +0000
Fix joins when using "NOT" for negations
Change: 4e103a8feb362f9b5722736c3fdda66d9f36819a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-05 20:20:40 +0000
Fixup broken query strings without joining keywords
This addresses the most used case where a query_string is missing the
keyword 'AND' in the token joining. This also makes it easier to produce
longer queries with less thinking.
Change: 1b43ea3120ae6ff51bc9f0d7c04e46f05499e7bb
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-02 23:13:54 +0000
Fix documentation for the ::Ranges plugin
Change: c7cb7c238a01c235823161c7d4455afe00665870
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-02 22:58:21 +0000
Add a --filter option to es-search.pl
Make use of the newly exposed 'filter' context from ::QueryString and add
positive matches as either 'must' (query context) or 'filter' (filter
context).
Change: 498af1d1759cf2788a06526a847d331e81e028d6
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-02 22:55:00 +0000
Allow toggling the context between 'query' and 'filter'
By overriding the ::QueryString context, it's possible to toggle between
collecting the positive matches as 'filter' elements instead of 'must'.
This allows the use of unscored queries which may improve performance for
some queries dramatically.
Change: 601bb7eea49cc3a8dbe0c9579c6a0b94e5675824
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-02 22:53:07 +0000
Early return for ::QueryString:: plugins
Both the ::Nested and ::Ranges plugins would emit uninitialized warnings
when run due to forgetting to check if the token matched the plugin.
Change: 67342d859955cc6f4ca4b6a75363beb08f7af1d8
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-02 21:41:22 +0000
Fix parameter specification in the library too
Change: 1039688ce0316e52f73f3ff57ec7b2ddedf0636b
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-02 21:36:26 +0000
Fix the parameter specifications and remove unsupported script
Change: 60aaa2dc1ea4ad19f0d5837a3620e9f98029196e
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-02 21:21:54 +0000
Add ::QueryString::Ranges
Translates queries with field:<,<=,>,>-= to equivalent range queries
Change: ca4e73f856bd9f48818e3fd2a7b959998dee8047
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-02 08:58:05 +0000
Fix --base in es-search.pl, make --index less policey.
* --base was broken as es-search.pl is looking for open indices.
Prepend the wildcard to the '/_stats/docs' request. * Don't validate the
--index parameter, instead pass it through. This
gives the user the power to create their own index strings
Change: a880e6ff664b99e64da8d18589f66bc51f1e089c
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-07-02 08:58:05 +0000
Replace es-storage-data.pl with es-storage-overview.pl
Change: 7bb0c293fd3f433183002c75be6a7e0e6e0f53fc
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-06-18 20:49:18 +0000
Remove size parameter if set to zero.
For statistical aggregations, the size parameter is invalid. This is a
shortcut to remove the parameter from the sub aggregation.
Change: 1e6372612b5e86627f6df2703c8a49e344bd129d
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-06-08 16:13:52 +0000
Fix failing tests
Change: 3053cd3ee7c1f30138f5fe2f51b6e77dc6639bba
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-05-10 21:40:51 +0000
Make the fast things fast
* Reorder the --bases for es-search.pl to skip index meta checks. * Use
es_indices(check_date => 0) instead of _all => 1 to skip closed
indices. * Make es_index_bases() simpler, i.e.:
Old behavior:
(a, a-b, a-b-c, b-c, c) = es_index_bases('a-b-c');
New behavior:
(a, a-b, a-b-c) = es_index_bases('a-b-c');
* Use _stats/docs to resolve all index names as indexes without explicit
aliases don't show up in all versions of ElasticSearch.
Change: ce08b1c0cc3c9432780612d13dd1214dc60be04e
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-05-10 21:06:34 +0000
New script: es-storage-overview.pl
Implements the simplest possible overview of storage. Does not provide node
or index data, but gives a high level overview.
Change: 3837060be0e0145b2e23987da6fab11af9d54a4e
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-05-10 20:29:24 +0000
es-search.pl - more concise output
When we encounter a record that is an array, but is holding only 1 item,
derefernce and extract the single value.
Change: b2473b3b412444a8148dfd96b0a07eb0dc466054
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-04-30 19:57:37 +0000
Made es_request() errors fatal and sped up index lookup
----------------------------------------
version 5.6 at 2018-04-13 18:20:03 +0000
----------------------------------------
Change: fcb981c7ea59b5e3ac5f9c5b2ae3cf34e1fad5a3
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-04-13 20:20:03 +0000
Documentation fixup with the latest pod tooling
Change: 2d9fce9813a585ff3115bfdce4b6b46a09192bf7
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-04-13 10:57:03 +0000
Don't track the local TODO file
Change: 7727fd81af17556c54e9c911d507efcd20266765
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-04-13 10:54:44 +0000
Ready for 5.6 Release
* Remove overzealous debugging messages * Tweak es-storage-data.pl to
provide full details via --all * Change the way
es-daily-index-maintenance.pl complains when --master--only is set. *
Documentation updated
Change: 1b06f8ce6a574e34acea51938193d7590c6c24f9
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-03-17 00:49:08 +0000
Docs updated added --view base to es-storage-data.pl
Change: 0cf82d70c23104b590f29b4551612578541cbb6a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-03-17 00:39:18 +0000
Get es-storage-data.pl working on modern ElasticSearch
----------------------------------------
version 5.5 at 2018-03-17 06:01:25 +0000
----------------------------------------
Change: 757dec9e648d077759da9a19ebdf3ed343df82f1
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-03-16 23:01:25 +0000
Remove no longer necessary test
Change: d63a1b823fcd72116e731cbfc45387290b60e769
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-03-16 22:49:15 +0000
Documentation updated for 5.5 release
Change: b1709e428bf9031e99b196ed6c865b02535ddb66
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-03-16 22:43:04 +0000
Bugfix and better replica management
* BUGFIX: es_index_strip_date() now removes everything after the date
string.
es-daily-index-maintenance.pl
* --replicas-* options aliased to --replica-* * Smarter handling of math
around replica calculation * Disable --bloom option entirely as it's
deprecated by Elastic
Change: aabba5ad9a1c26ee5aa631f4ea81d887bf98463a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-02-21 23:04:18 +0000
Pod and Perl::Critic fixes
Change: 1dcaaa7ea8b7b8b89387692dd969a18fd385bd23
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-02-21 20:58:15 +0000
Conversion to Ref::Util and Types::Tiny
Fix paths for the shebang line so the installer will be able to install the
scripts with the correct Perl binary path. Add a first pass at nested query
syntax
Change: 722e812a7365f9a81aa0d1df5d61a3a1dcb695ed
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2018-02-21 20:52:06 +0000
Update the QS::IP expander to make it more intelligent.
If a field ends in *_ip, allow CIDR queries against the field. This skips
renaming the field from "src_ip" to "src_ip_numeric" and is now an inline
replacement. This enhances simplifies the CLI queries.
The docs were not accurate, so now the example and documentation are valid.
Change: 8e7e34fe1690349a0485ec271c85e966c699b0d9
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-12-03 23:42:00 +0000
Disallow tabs in my files.
Change: 938d5f9e9bf387d304388bf524348dd4b0d51fac
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-10-25 17:07:47 +0000
Allow --show to be more flexible
* Unbind from the index fields since this operates on the source
document * Change to an array so you can add multiple --show's
----------------------------------------
version 5.4 at 2017-10-13 19:36:20 +0000
----------------------------------------
Change: c31b5f39953589c497f4374be9c22f36a2514dc2
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-10-13 12:36:20 +0000
Documentation update for the 5.4 release.
Change: 3b6a379937869eaf46e80eea6f7d54e5c4f2220a
Author: Brad Lhotsky <brad@divisionbyzero.net>
Date : 2017-07-12 20:50:57 +0000
Default to UTC, warn when using incompatible options.
Change: 570de7b2ffa8896b723a3d5c739cec25f4fe8297
Author: Lisa Hare <lhare@inview.co.uk>
Date : 2017-06-23 08:50:42 +0000
add older option to es-apply-settings
when older argument is passed to es_indices in Utilities.pm if days is also
specified then it will only return indexes older that the specified days
Refs #24
Change: 04a72068727474471abd709e993b9095bcf4e489
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2017-06-07 11:05:12 +0000
Fixup the optimize call for newer ElasticSearch releases.
Change: 25ad9e95697a92093c30375836041c7fb2f691e8
Author: Markus Linnala <Markus.Linnala@cybercom.com>
Date : 2017-04-05 19:33:52 +0000
es-alias-manager.pl: relax index name checker to allow _
----------------------------------------
version 5.3 at 2017-03-23 01:49:52 +0000
----------------------------------------
Change: 3becaead7ab98d420511e67ac55724eb19c3bd76
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2017-03-22 18:49:52 +0000
Version 5.3 Docs
Change: 2fb6c954347fbbd3a5c8dc372effe8e84116b908
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2017-03-22 18:48:01 +0000
ElasticSearch 5.2.x support and enhancements
* Elasticsearch no longer supports aliased keys in the index.
Removed the uniqueness parsing for the last section of the key name. ES
v2.x and on requires the keys be fully qualified.
* es-search.pl subaggs output adjusted
Prior version injected the sub aggregations before the values, this now
append the sub aggregations to the listed output in a more readable
manner.
* Allow aggregations to be wrapped and use it to provide date_histogram
wrappers for es-search.pl
es-search.pl now supports --interval to wrap aggregations in a
date_histogram's so you can better understand what is happening with
your data.
* es_master() and --master-only added, documentation clean up.
----------------------------------------
version 5.2 at 2016-12-29 22:44:10 +0000
----------------------------------------
Change: f9be6c93da4bb0d734440c5f5ccabef81888adc2
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-12-29 14:44:10 +0000
Fixes for 5.x
* <index>/_status deprecated since 1.2, make rewrite to _stats the
default * <index>/_stats no longer supports all parameter, it's the default
since 1.x * Replace the last call to <index>/_status in my tools with
<index>/stats
----------------------------------------
version 5.1 at 2016-11-16 02:30:42 +0000
----------------------------------------
Change: f20cbf1927d19c4aeac2c1bebf792cd77e4f7986
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-11-15 18:30:42 +0000
Added perl 5.24 testing to travis and bumped version numbers.
Change: df93facfcc57614bc9ed011a6dadea4a0510f832
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-11-15 18:26:35 +0000
Improvements to the UI and the compatibility layer
* es-search.pl now supports sub aggregations using the --with parameter *
Fix a bug in the handling of es_indices with ES 2.x clusters. * Remove the
warning from the App::ElasticSearch::Utilities::HTTPRequest
object as I can see this moving out of this module soon. *
App::ElasticSearch::Utilities::Connection handles JSON error documents *
Fix case sensitivity bug in the --top option of es-search.pl
----------------------------------------
version 5.0 at 2016-08-19 03:50:24 +0000
----------------------------------------
Change: a8d6911a4ae7238dd564d4235a35c52528adb19a
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-18 20:50:24 +0000
Documentation re-generated for 5.0 release.
Change: 94a5189553fb6c070f2156b4e1fd173745cbb81e
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-18 18:03:47 +0000
VersionHacks now works on x.0 releases.
Previous releases failed to compare the string of the version because Perl
was helpfully casting '5.0' to '5' as the key in the vhacks hashes. This is
now solved by assembling a version string.
Change: 3c58599fc9972fdaefeb9b14e62a15d711a68a03
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-18 17:49:59 +0000
es-graphite-dynamic.pl collector compatibility.
Allow this to be called from diamond or collectd as a script collector with
prefix overrides.
Change: 6437ab2119af9051d1ab14eff40ef90c4bb0442f
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-18 17:23:21 +0000
Preparing for 5.0 release.
Added es_pass_exec() feature to allow for storing passwords in keychains or
password vaults instead of the command line or a plaintext file.
Extensive documentation updates to cover everything the module does and how
the arguments work with examples where required.
Change: fd13349f01f6312272b0a78a90141ce057d29715
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-18 13:38:00 +0000
Hook basic auth and protocol into App::ElasticSearch::Utilities.
The basic auth handler uses the monkey patching tweak by redefining
LWP::UserAgent::get_basic_credentials() to call the
App::ElasticSearch::Utilities::es_basic_auth() function. This makes the
whole code base cleaner as we don't have to bother with authentication
**unless** the server requires it.
Authentication is resolved by looking at the username/password specified by
--http-username and --http-password, then by the YAML config files
(/etc/es-utils.yaml, ~/.es-utils.yaml), then using the netrc files, then as
a last ditch, prompting the user for the password on the CLI.
If CLI::Helpers v1.1 or greater is installed, the password prompts will
disable echo and not perform ReadLine magic.
HTPS is supported via the --proto option. Strict host verification is the
default. Dear lord, please stop with the setting host verify 'false'. This
is why we can't have nice things. I'm not supporting that option at this
time, or ever. Fix your certificates, TYVM.
Change: f54615b587f3ce92eef515814c043e838bace5c4
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-18 10:44:52 +0000
Fix insanity with determining the version number with a string eval().
Change: ee3cbbe270018c532f566483e4684c3637150957
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-18 10:33:59 +0000
Merge branch 'feature/use_lwp' into pre5.0
Change: db7cb5b5327162f53cb640cee36d6eb0e7583685
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-17 20:26:49 +0000
Docs updated for the --with element in es-search.pl
Change: 6c14ac9a079e5fc9d5e11ce98dfd58925226282f
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-17 20:25:00 +0000
Added --with <agg>:<field>:<size> to es-search.pl
Sub-aggregations using --with are now supported to help make more sense out
of the data from --top.
Change: bad8827e666ede37f828d066496446abffcbf106
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-17 20:01:35 +0000
Starting the Basic Auth pass through bits.
Change: 10ed9d4341b44f51c9875e6f83b5761654ef5d8a
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-17 17:03:59 +0000
App::ElasticSearch::Utilities::HTTPRequest added.
There are a few places where a standard HTTP::Request object, or even an
HTTP::Request::JSON objects will barf on the payloads that we need to send
to Elasticsearch. A::ES::U::HTTPRequest subclasses HTTP::Request and
overrides the new() and content() methods.
* new() - Adds the Accept: application/json header.
* content() - Handle how perl datastructures are encoded in the body.
* HASH: Hashes are JSON encoded
* ARRAY: Array elements are encoded as json and then those are
joined together with "\n" (think the /_bulk API)
* Non-reference Scalars: passed as is, you know what you're doing.
* Everything else: discarded, generates a warning
The '_cat' API passes JSON back when you tell it you can read that, so
updated the es_indices() function to handle the JSON resulting from the
_cat call using the new A::ES::U::HTTPRequest object.
Change: 3cf2c13f87a911b88c02297ecbf280f456e61bf3
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-08-03 17:26:08 +0000
Working version of the toolkit using LWP::UserAgent instead of
Hijk/Elastijk
Change: 176290d6ed44dba88db43eb5a3541fd981fca602
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-07-31 22:31:41 +0000
Use the App::ElasticSearch::Utilities::Connection object instead of
Hijk/Elastijk. Start migration to Ref::Util
Change: d5a78e1db33abe3d83ca86e1b178f1fdd5b519eb
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-07-31 22:30:07 +0000
Connection class to abstract the HTTP bits from this module.
Change: 2eea122cf0732ac98c6f50ae67377ad0d538592a
Author: Daniel Ostermeier <daniel.ostermeier@gmail.com>
Date : 2016-06-20 10:52:55 +0000
Change: e0b4d0e7fdf03cabab9f5b78406b1726c994a69a
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-05-04 15:21:12 +0000
Handle nested objects in the --show option.
Change: 6049eeb58495a2aa7dfcaffae876a2ae6e478958
Author: Brad Lhotsky <blhotsky@craigslist.org>
Date : 2016-04-07 14:20:28 +0000
Version Numbers Updated
=================================================
Plus 56 releases after 2016-07-21 00:00:00 +0000.
=================================================