NAME

Robots::Validate - Validate that IP addresses are associated with known robots

VERSION

version v0.3.1

SYNOPSIS

use Robots::Validate;

my $rv = Robots::Validate->new;

...

if ( my $res = $rs->validate( $ip, $user_agent ) ) {
   ...
}

DESCRIPTION

This module allows one to validate a robot user-agent string against the IP addresses.

ATTRIBUTES

resolver

This is the Net::DNS::Resolver object used for DNS lookups.

This can only be set via the constructor.

networks

This is a Net::Patricia object used for matching networks.

This can only be set via the constructor.

Note that internally IPv4 addresses are converted to IPv6 addresses.

config

This is an array reference of rule configurations. Each item is a hash reference with the following keys:

name

This is a short string with the rule name.

agents

This is an array reference of short strings to match against user agent strings. It is required.

domain

This is a string or array reference of short strings with the domain suffix. e.g. .crawl.example.com, or with a regular expression /\.crawl\.example\.com$/.

network

This is an optional array reference of CIDR network blocks.

match

This specifies the match type.

The possible values are:

any

An agent is verified if either the domain or the network match. This is the default when unspecified.

all

An agent is verified is both the domain and the network match.

If the constructor is passed a hash reference, then it is coerced into an array reference of the values, sorted by keys, where the key is added to the name if it is not already specified. (The agents and network values will be coerced into array references.)

If the constructor is passed anything else, it is assumed to be the filename of a TOML file with the configuration.

index

This is a hash reference where the keys are rule names and the values are the rules from "config".

locked

This is a boolean to indicate that internal data structures for matching agents have been built, and the rules are locked.

METHODS

validate

my $result = $rv->validate( $ip, $agent, \%opts );

Alternatively, you can pass in a Plack environment:

my $result = $rv->validate($env);

This method attempts to validate that an IP address $ip is associated with a known robot identified by the $agent.

If $ip is in a known list of network blocks, then it succeeds. Otherwise it attempts to validate that an IP address belongs to a known robot by first looking up the hostname that corresponds to the IP address, and then validating that the hostname resolves to that IP address. It then checks if the hostname is associated with a known web robot.

If that succeeds, it returns an array reference containing the name and the matching agent string.

The rule can be looked up from the "index" attribute.

You can specify the following %opts:

no_cache

Do not check the "cache".

agent

Specify the $agent, for backwards-compatibility with versions before v0.3.0.

This is deprecated and will be removed from a future version.

cache

This is an optional CHI cache used for matching IP addresses and user agent strings.

has_cache

This indicates that there is a "cache".

cache_options

This is an optional hash reference of "cache" options to pass to "compute" in CHI, e.g.

{ expires_in => '8 hours' }

Plain strings are assumed to be expires_in values.

KNOWN ISSUES

Many of these rules are not documented, but have been guessed from web traffic.

The networks used by some robots do not consistently support reverse DNS lookups, and may randomly fail.

SECURITY CONSIDERATIONS

When using the "cache", ensure that it is configured to expire the data and digest the keys by setting "max_key_length" in CHI to 0. This is to keep the cache from growing too large, and to reduce the likelihood of cache backend vulnerabilities being exploited through user-agent strings.

SEE ALSO

The file robots.toml included with this distribution contains links to documented rules.

SOURCE

The development version is on github at https://github.com/robrwo/Robots-Validate and may be cloned from https://github.com/robrwo/Robots-Validate.git

SUPPORT

Only the latest version of this module will be supported.

This module requires Perl v5.24 or later, based on the minimum Perl supported by Dist::Zilla.

Reporting Bugs and Submitting Feature Requests

Please report any bugs or feature requests on the bugtracker website https://github.com/robrwo/Robots-Validate/issues

When submitting a bug or request, please include a test-file or a patch to an existing test-file that illustrates the bug or desired feature.

If the bug you are reporting has security implications which make it inappropriate to send to a public issue tracker, then see SECURITY.md for instructions how to report security vulnerabilities.

AUTHOR

Robert Rothenberg <perl@rhizomnic.com>

Some of the development of this module was sponsored by Science Photo Library https://www.sciencephoto.com.

CONTRIBUTOR

Robert Rothenberg <rrwo@cpan.org>

COPYRIGHT AND LICENSE

This software is Copyright (c) 2018-2026 by Robert Rothenberg.

This is free software, licensed under:

The Artistic License 2.0 (GPL Compatible)