/ 
Robots-Validate-v0.3.7-TRIAL
⭐ Starred 0 / Robots::Validate

NAME

Robots::Validate - Validate that IP addresses are associated with known robots

VERSION

version v0.3.7

SYNOPSIS

use Robots::Validate;

my $rv = Robots::Validate->new;

...

if ( my $res = $rs->validate( $ip, $user_agent ) ) {
   ...
}

DESCRIPTION

This module allows one to validate a robot user-agent string against the IP addresses.

ATTRIBUTES

resolver

This is the Net::DNS::Resolver object used for DNS lookups.

This can only be set via the constructor.

networks

This is a Net::Patricia object used for matching networks.

This can only be set via the constructor.

Note that internally IPv4 addresses are converted to IPv6 addresses.

config

This is an array reference of rule configurations. Each item is a hash reference with the following keys:

name

This is a short string with the rule name. It is required.

This name will be transformed into lowercase.

agents

This is an array reference of short strings to match against user agent strings. It is required.

Case will be ignored, so "GoogleBot" and "googlebot" are the same.

It is important to use strings that are unique and not substrings or superstrings of other user agent strings. Otherwise robot validation may fail if there are conflicting matches with the wrong user agent.

domain

This is a string or array reference of short strings with the domain suffix. e.g. .crawl.example.com, or with a regular expression /\.crawl\.example\.com$/.

It is important that domain suffixes begin with an initial dot. For cases where an entire domain name should match, use a regular expression that anchors the beginning of the string, e.g. /^crawl\.example\.com$/. Otherwise an imposter domain matching the dotless-suffix would be validated, e.g. imposter-crawl.example.com.

Also note that the TOML format will require slashes to be escaped, e.g.

domain = "/\\.google(bot)?\\.com$/"
network

This is an array reference of CIDR network blocks.

match

This specifies the match type.

The possible values are:

any

An agent is verified if either the domain or the network match. This is the default when unspecified.

all

An agent is verified is both the domain and the network match.

Note that either domain or network can be omitted.

If the constructor is passed a hash reference, then it is coerced into an array reference of the values, sorted by keys, where the key is added to the name if it is not already specified. (The agents and network values will be coerced into array references.)

If the constructor is passed anything else, it is assumed to be the filename of a TOML file with the configuration. The file will be parsed with TOML::XS if it is available, or TOML::Tiny otherwise.

There is a utility in the distribution devel/rebuild-robots-config that will normalise the file and update network information about robots.

Users are encouraged to copy the share/robots.toml file from the distribution and maintain a separate file with rules that are suited to their application.

Users are also encouraged to submit new and updated rules back to the maintainers. See "SOURCE".

index

This is a hash reference where the keys are rule names and the values are the rules from "config".

locked

This is a boolean to indicate that internal data structures for matching agents have been built, and the rules are locked.

cache

This is an optional CHI cache used for matching IP addresses and user agent strings.

See the "SECURITY CONSIDERATIONS" section for improving the safety of the cache.

has_cache

This indicates that there is a "cache".

cache_options

This is an optional hash reference of "cache" options to pass to "compute" in CHI, e.g.

{ expires_in => '8 hours' }

Plain strings are assumed to be expires_in values.

METHODS

validate

my $result = $rv->validate( $ip, $agent, \%opts );

Alternatively, you can pass in a Plack environment:

my $result = $rv->validate($env);

This method attempts to validate that an IP address $ip is associated with a known robot identified by the $agent.

If $ip is in a known list of network addresses, then it succeeds.

Otherwise it attempts to validate that an IP address belongs to a known robot by first looking up the hostname that corresponds to the IP address, and then validating that the hostname resolves to that IP address. It then checks if the hostname is associated with a known web robot.

A fake robot (one where the user agent claims to be something that does not match the IP address or resolved hostname) returns a defined but false value.

An unknown user agent returns undef.

Successful checks return an array reference containing the name and the matching agent string.

The rule can be looked up from the "index" attribute.

You can specify the following %opts:

no_cache

Do not check the "cache".

cache_failure

By default, failures are not cached.

When this is set to true, failures are cached.

Any value other than "1" is assumed to be an "expires_in" in CHI option for setting the cache.

See "SECURITY CONSIDERATIONS" before enabling this feature.

This has no meaning if there is no cache or the no_cache option is set.

This was added in version v0.3.4. This can also be specified in the "cache_options" as of v0.3.5.

agent

Specify the $agent, for backwards-compatibility with versions before v0.3.0.

This is deprecated and will be removed from a future version.

bad_robot

$rv->bad_robot( $env, \%opts ) and ...

This is a wrapper around "validate" that returns true for imposter robots.

It will return undef when the result is unknown.

KNOWN ISSUES

Many of these rules are not documented, but have been guessed from web traffic.

Bots that use cloud services without documenting what hosts they use are not added here.

The networks used by some robots do not consistently support reverse DNS lookups, and may randomly fail.

SECURITY CONSIDERATIONS

When using the "cache", ensure that it is configured to expire the data by setting "expires_in" in CHI and digest the keys by setting "max_key_length" in CHI to 0. This is to keep the cache from growing too large, and to reduce the likelihood of cache backend vulnerabilities being exploited through user-agent strings.

When setting the cache_failure option, be aware that cached failures may need a shorter expiration time.

When specifying a domain for verification rules, ensure that there is an initial dot in the suffix, or that the regular expression matches the entire domain name. Otherwise imposter domains with the same suffix will be validated.

When the network list contains cloud addresses, it is important to regularly update the addresses from the documented information, as an imposter can use abandoned cloud IP addresses.

SEE ALSO

The file robots.toml included with this distribution contains links to documented rules.

The TOML specification can be found at https://toml.io.

SOURCE

The development version is on github at https://github.com/robrwo/Robots-Validate and may be cloned from https://github.com/robrwo/Robots-Validate.git

SUPPORT

Only the latest release of this module will be supported.

This module requires Perl v5.24 or later. Future releases may only support Perl versions released in the last ten (10) years.

Reporting Bugs and Submitting Feature Requests

Please report any bugs or feature requests on the bugtracker website https://github.com/robrwo/Robots-Validate/issues

When submitting a bug or request, please include a test-file or a patch to an existing test-file that illustrates the bug or desired feature.

If the bug you are reporting has security implications which make it inappropriate to send to a public issue tracker, then see SECURITY.md for instructions how to report security vulnerabilities.

AUTHOR

Robert Rothenberg <perl@rhizomnic.com>

Some of the development of this module was sponsored by Science Photo Library https://www.sciencephoto.com.

COPYRIGHT AND LICENSE

This software is Copyright (c) 2018-2026 by Robert Rothenberg.

This is free software, licensed under:

The Artistic License 2.0 (GPL Compatible)