Search::ElasticDirect - An interface to ElasticSearch API, version independent
version 2.5.0
use Search::ElasticDirect; my $elk = Search::ElasticDirect->new( host=> 'SomeServer', protocol=> 'http', authentication=> 'no' ); die "Connection error because $elk->{error}\n" if $elk->{error}; my $reply = $elk->direct( method=>'GET', path=>'_cat/nodes', param=>'h=name,ip,role,version&s=name:asc', format=>'data' ); die "$elk->{error}\n" if $elk->{error}; die "Allowed to run only at active master\n" unless $elk->ActiveMaster; die $elk->{error} if $elk->{error}; foreach my $srv (@{$reply}) { say join ',', $srv->{name}, $srv->{ip}, $srv->{version}, $srv->{role} }
An interface to ElasticSearch API, version independent
The DeepPagination methods are added to help with complexity of the scroll/cursor/search_after APIs
Check for errors the property $obj->{error} . if everything is ok it is FALSE
Create a new ElasticDirect object and define the connection parameters to the elasticsearch cluster.
Most of the times you will use this one
my $elk = Search::ElasticDirect->new( host => 'SomeServer', port => 9200, protocol => 'https', timeout => 1800, keep_alive => 'yes', authentication => 'yes', username => 'Joe', password => 'JoePass', certca => '/etc/elasticsearch/chain.pem', verify_SSL => 'no' ); die $elk->{error} if $elk->{error};
If you are using the SearchGuard security plugin and you need node admin key/cert without username and password to perform admin tasks e.g. to delete an immutable index
my $elk = Search::ElasticDirect->new( host => 'SomeServer', port => 9200, protocol => 'https', timeout => 1800, keep_alive => 'yes', authentication => 'no', certca => '/etc/elasticsearch/chain.pem', cert => '/etc/elasticsearch/SomeServer.pem', key => '/etc/elasticsearch/SomeServer.key', verify_SSL =>'no' ); die $elk->{error} if $elk->{error};
Without certificate
my $elk = Search::ElasticDirect->new( host => 'SomeServer', port => 9200, protocol => 'https', timeout => 1800, keep_alive => 'yes', authentication => 'yes', username => 'Joe', password => 'JoePass' ); die $elk->{error} if $elk->{error};
Plain text with authentication
my $elk = Search::ElasticDirect->new( host => 'SomeServer', port => 9200, protocol => 'http', timeout => 1800, keep_alive => 'yes', authentication => 'yes', username => 'Joe', password => 'SomePass' ); die $elk->{error} if $elk->{error};
Plain text without authentication
my $elk = Search::ElasticDirect->new( host => 'SomeServer', port => 9200, protocol => 'http', timeout => 1800, keep_alive => 'yes', authentication => 'no' ); die $elk->{error} if $elk->{error};
If you have schedule your script at multiple servers for high availability but it must run once at active master server
my $obj = Search::ElasticDirect->new(...); die $elk->{error} if $elk->{error}; die "Allowed to run only at active master\n" unless $elk->ActiveMaster; die $elk->{error} if $elk->{error};
Pass through the request to ElasticSeach API transparent without much validation, so you must know the correct syntax and parameters of the API/path you are calling
You can receive the result at the format data, yaml, json, json_pretty, txt
The data format returns a Perl data structure e.g. hash or array, while the other are text formatted as defined
Examples
my $reply = $elk->direct(method=>'GET', path=>'_cat/health', param=>'filter_path=status', format=>'yaml'); die $elk->{error} if $elk->{error}; say $reply
$elk->direct(method=>'GET', path=>'_cat/health', param=>'filter_path=status', format=>'json_pretty');
$reply = $elk->direct(method=>'GET', path=>'_cat/health', format=>'data'); say $reply->[0]->{status};
$reply = $elk->direct(method=>'GET', path=>'_cat/indices', param=>'h=index,health,docs.count,store.size&bytes=g&s=index:asc', format=>'data'); die $elk->{error} if $elk->{error}; foreach ( @{ $reply } ) { say "$_->{index}, $_->{health}, $_->{'docs.count'}, $_->{'store.size'}" }
$reply = $elk->direct( method => 'GET', path => 'foo/_search', format => 'data', send_data => '{"size": 2, "query": {"range": {"@timestamp": {"from": "1970-01-01T00:00:00.000Z", "to": "2036-01-03T23:59:59.999Z"}}}}' ); die $elk->{error} if $elk->{error}; foreach ( @{ $reply->{hits}->{hits} } ) { say say "$_->{_source}->{'@timestamp'} , $_->{_source}->{id}" }
format can be here data, txt, csv, yaml, json, json_pretty, cbor, smile
$reply = $elk->direct( method => 'POST', path => '_sql', format => 'data', send_data => "{\"query\": \"SELECT * FROM foo WHERE id >= '01' ORDER BY id DESC LIMIT 100\"}" ); die $elk->{error} if $elk->{error}; if ($elk->{format} eq 'data') { foreach ( @{$reply->{rows}} ) { say join ' , ', @{$_} } } else { say $reply }
Retrieve massive amount of results using the scroll API
documents here is how many records to fetch per iteration
format can be data, yaml, json or json_pretty
my $iterator = $elk->DeepPagination_scroll( index => 'foo', documents => 9000, format => 'data', send_data => '{"query": {"range": {"@timestamp": {"from": "1970-01-01T00:00:00.000Z", "to": "2036-01-03T23:59:59.999Z"}}}}', scroll => '10m' ); die $elk->{error} if $elk->{error}; while (my $Documents = $iterator->()) { if ($elk->{format} eq 'data') { foreach my $doc (@{$Documents}) { foreach my $key (sort keys %{$doc->{_source}}) { say "$key : $doc->{_source}->{$key}"; } } } else { say $Documents } }
Retrieve massive amount of results using the search_after API
If you choose to use the PIT (point in time) it will created automatic for you
At sort, define as less fields you can that produce a unique combination. The fields must be sortable reasonable.
my $iterator = $elk->DeepPagination_search_after( index => 'foo', PIT => 'yes', keep_alive=> '10m', documents => 9000, send_data => '{ "query": {"range": {"@timestamp": {"from": "1970-01-01T00:00:00.000Z", "to": "2036-01-03T23:59:59.999Z"}}} }', sort => [ {'@timestamp'=> {order=> 'asc', format=> "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"}} , {id=>'asc'} ], ); while (my $Documents = $iterator->()) { foreach my $doc (@{$Documents}) { foreach my $key (sort keys %{$doc->{_source}}) { say "$key : $doc->{_source}->{$key}" } } }
Retrieve massive amount of results using the sql API
format can be data, csv, txt, yaml, json, json_pretty, cbor, smile
my $iterator = $elk->DeepPagination_sql( documents => 100, format => 'data', send_data => '{ "query": "SELECT * FROM foo ORDER BY id DESC" }' ); die $elk->{error} if $elk->{error}; while (my $Documents = $iterator->()) { if ($elk->{format} eq 'data') { use Data::Dumper; say Dumper $Documents } else { print $Documents } }
Search::Elasticsearch The official client for Elasticsearch
Dancer2::Plugin::ElasticSearch ElasticSearch wrapper for Dancer
Any ideas or contributors are welcome
George Bouras <george.mpouras@yandex.com>
This software is copyright (c) 2023 by George Bouras.
This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.
To install Search::ElasticDirect, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Search::ElasticDirect
CPAN shell
perl -MCPAN -e shell install Search::ElasticDirect
For more information on module installation, please visit the detailed CPAN module installation guide.