NAME

Database::BI::SafeUA - Fetch a URL only if it leads to the public internet

VERSION

0.023.1

SYNOPSIS

my $ua  = Database::BI::SafeUA->new(private_hosts => ['intranet.example.com']);
my $res = $ua->get('https://example.com/table.html');
# http://localtest.me/ -> 403 "Refused: localtest.me is 127.0.0.1, which is this machine"

DESCRIPTION

An LWP::UserAgent for url: sources and /import. Every request, including each redirect (at most 5), is resolved, refused with a 403 response if any address is on this machine, a private or shared network, link-local (cloud metadata services), multicast or reserved, and then sent to the address that was checked. The name stays in the Host header, the TLS server name (SNI) and the certificate check, so DNS that answers differently on a second lookup (rebinding) changes nothing. An IPv4 address inside IPv6 (mapped, NAT64, compatible, 6to4) is checked as that IPv4 address. Only http and https are fetched.

METHODS

new

API SPECIFICATION

INPUT

private_hosts: arrayref of host names, optional; names (or *.suffix) that
               may resolve to private addresses (an intranet).  An address
               written as such is never one of them.
resolve:       code ref, optional; host -> list of address strings (tests)
anything else: as LWP::UserAgent->new (timeout 30, max_redirect 5 by default)

OUTPUT

The user agent.

send_request

As "send_request" in LWP::UserAgent, with the checks above. LWP calls it for every request and each redirect; callers use get and the rest as usual.

API SPECIFICATION

INPUT

$request: HTTP::Request
$arg, $size: as LWP::UserAgent (content file or callback, read size)

OUTPUT

An HTTP::Response whose request is the one given (not the pinned copy). A refused request gets 403 Refused: <why> and Client-Warning: Internal response, and nothing is sent: a scheme other than http or https, no host, a name that does not resolve, or any address blocked_address refuses (unless private_host_allowed allows the name).

private_host_allowed

$ua->private_host_allowed('reports.corp.example');   # 1 with private_hosts => ['*.corp.example']

API SPECIFICATION

INPUT

$host: a host name (case does not matter)

OUTPUT

1 when private_hosts names it, or a *.suffix entry ends it (*.corp.example matches a.corp.example, not corp.example or evilcorp.example), else 0. send_request asks this only for a name, never for an address written as one.

FUNCTIONS

blocked_address

my $why = Database::BI::SafeUA::blocked_address('10.1.2.3');   # 'a private network'

API SPECIFICATION

INPUT

$ip: an IPv4 or IPv6 address as text (brackets and a zone index are allowed)

OUTPUT

Why the address is refused, or undef when it is public. Text that is not an address is refused.

numeric_host

numeric_host('2130706433');   # '127.0.0.1'
numeric_host('example.com');  # undef

API SPECIFICATION

INPUT

$host: a URL's host

OUTPUT

The address the host is when it is written as one, in any form the resolver reads (127.1, 0x7f000001, 0177.0.0.1, [::1]), else undef.

resolve_host

API SPECIFICATION

INPUT

$host: a host name

OUTPUT

Its addresses (getaddrinfo), or an empty list.

AUTHOR

Nigel Horne <njh@nigelhorne.com>

LICENCE AND COPYRIGHT

Copyright 2026 Nigel Horne.

Usage is subject to the GPL2 licence terms.