The London Perl and Raku Workshop takes place on 26th Oct 2024. If your company depends on Perl, please consider sponsoring and/or attending.

NAME

Net::Google::SafeBrowsing4 - Perl extension for the Google Safe Browsing v4 API.

SYNOPSIS

	use Log::Log4perl qw(:easy);
	use Net::Google::SafeBrowsing4;
	use Net::Google::SafeBrowsing4::Storage::File;

        Log::Log4perl->easy_init($DEBUG);

	my $storage = Net::Google::SafeBrowsing4::Storage::File->new(path => '.');
	my $gsb = Net::Google::SafeBrowsing4->new(
		key 	=> "my key",
		storage	=> $storage,
		logger	=> Log::Log4perl->get_logger(),
	);

	$gsb->update();
	my @matches = $gsb->lookup(url => 'http://ianfette.org/');

	if (scalar(@matches) > 0) {
		print("http://ianfette.org/ is flagged as a dangerous site\n");
	}

	$storage->close();

DESCRIPTION

Net::Google::SafeBrowsing4 implements the Google Safe Browsing v4 API.

The Google Safe Browsing database must be stored and managed locally. Net::Google::SafeBrowsing4::Storage::File uses files as the storage back-end. Other storage mechanisms (databases, memory, etc.) can be added and used transparently with this module.

The source code is available on github at https://github.com/juliensobrier/Net-Google-SafeBrowsing4.

If you do not need to inspect more than 10,000 URLs a day, you can use Net::Google::SafeBrowsing4::Lookup with the Google Safe Browsing v4 Lookup API which does not require to store and maintain a local database.

IMPORTANT: Google Safe Browsing v4 requires an API key from Google: https://developers.google.com/safe-browsing/v4/get-started.

CONSTANTS

Several constants are exported by this module:

DATABASE_RESET

Google requested to reset (empty) the local database.

INTERNAL_ERROR

An internal error occurred.

SERVER_ERROR

The server sent an error back to the client.

NO_UPDATE

No update was performed, probably because it is too early to make a new request to Google Safe Browsing.

NO_DATA

No data was sent back by Google to the client, probably because the database is up to date.

SUCCESSFUL

The update operation was successful.

CONSTRUCTOR

new()

Create a Net::Google::SafeBrowsing4 object

my $gsb = Net::Google::SafeBrowsing4->new(
	key	=> "my key",
	storage	=> Net::Google::SafeBrowsing4::Storage::File->new(path => '.'),
	lists	=> ["*/ANY_PLATFORM/URL"],
);

Arguments

base

Safe Browsing base URL. https://safebrowsing.googleapis.com by default

key

Required. Your Google Safe Browsing API key

storage

Required. Object which handles the storage for the Google Safe Browsing database. See Net::Google::SafeBrowsing4::Storage for more details.

lists

Optional. The Google Safe Browsing lists to handle. By default, handles all lists.

logger

Optional. Log::Log4perl compatible object reference. By default this option is unset, making Net::Google::SafeBrowsing4 silent.

perf

Optional. Set to 1 to enable performance information logging. Needs a logger, performance information will be logged on DEBUG level.

version

Optional. Google Safe Browsing version. 4 by default

http_agent

Optional. LWP::UserAgent to use for HTTPS requests. Use this option for advanced networking options, like proxies or local addresses.

http_timeout

Optional. Network timeout setting for LWP::UserAgent (60 seconds by default)

http_compression

Optional. List of accepted compressions for HTTP response. Enabling all supported compressions reported by HTTP::Message by default.

max_hash_request

Optional. maximum number of full hashes to request. (500 by default)

PUBLIC FUNCTIONS

update()

Performs a database update.

$gsb->update();

Returns the status of the update (see the list of constants above): INTERNAL_ERROR, SERVER_ERROR, NO_UPDATE, NO_DATA or SUCCESSFUL

This function can handle multiple lists at the same time. If one of the lists should not be updated, it will automatically skip it and update the other one. It is faster to update all lists at once rather than doing them one by one.

Arguments

lists

Optional. Update specific lists. Use the list(s) from new() by default. List are in the format "MALWARE/WINDOWS/URLS" or "*/WINDOWS/*" where * means all possible values.

force

Optional. Force the update (1). Disabled by default (0).

Be careful if you set this option to 1 as too frequent updates might result in the blacklisting of your API key.

get_lists()

Gets all threat list names from Google Safe Browsing and save them.

my $lists = $gsb->get_lists();

Returns an array reference of all the lists:

[
	{
		'threatEntryType' => 'URL',
		'threatType' => 'MALWARE',
		'platformType' => 'ANY_PLATFORM'
	},
	{
		'threatEntryType' => 'URL',
		'threatType' => 'MALWARE',
		'platformType' => 'WINDOWS'
	},
	...
]

or C<undef> on error. This method updates C<$gsb->{last_error}> field.

lookup()

Looks up URL(s) against the Google Safe Browsing database.

Returns the list of hashes along with the list and any metadata that matches the URL(s):

(
	{
		'lookup_url' => '...',
		'hash' => '...',
		'metadata' => {
			'malware_threat_type' => 'DISTRIBUTION'
		},
		'list' => {
			'threatEntryType' => 'URL',
			'threatType' => 'MALWARE',
			'platformType' => 'ANY_PLATFORM'
		},
		'cache' => '300s'
	},
	...
)

Arguments

lists

Optional. Lookup against specific lists. Use the list(s) from new() by default.

url

Required. URL to lookup.

PRIVATE FUNCTIONS

These functions are not intended to be used externally.

make_lists()

Transforms a list from a string expression (eg.: "MALWARE/*/*") into a list object.

update_error()

Handle server errors during a database update.

make_lists_for_update()

Formats the list objects for update requests.

request_full_hash()

Requests full full hashes for specific prefixes from Google.

parse_full_hashes()

Processes the request for full hashes from Google.

PROXIES AND LOCAL ADDRESSES

To use a proxy or select the network interface to use, simply create and set up an LWP::UserAgent object and pass it to the constructor:

use LWP::UserAgent;
use Net::Google::SafeBrowsing4;
use Net::Google::SafeBrowsing4::Storage::File;

my $ua = LWP::UserAgent->new();
$ua->env_proxy();

# $ua->local_address("192.168.0.14");

my $gsb = Net::Google::SafeBrowsing4->new(
	key		=> "my-api-key",
	storage		=> Net::Google::SafeBrowsing4::Storage::File->new(path => "."),
	http_agent	=> $ua,
);

Note that the Net::Google::SafeBrowsing4 object will override certain LWP properties:

timeout

The network timeout will be set according to the http_timeout constructor parameter.

Content-Type

The Content-Type default header will be set to application/json for HTTPS Requests.

Accept-Encoding

The Accept-Encoding default header will be set according to the http_compression constructor parameter.

SEE ALSO

See Net::Google::SafeBrowsing4::URI about URI parsing for Google Safe Browsing v4.

See Net::Google::SafeBrowsing4::Storage for the list of public functions.

See Net::Google::SafeBrowsing4::Storage::File for a back-end storage using files.

Google Safe Browsing v4 API: https://developers.google.com/safe-browsing/v4/

AUTHOR

Julien Sobrier, <julien@sobrier.net>

COPYRIGHT AND LICENSE

Copyright (C) 2017 by Julien Sobrier

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.8 or, at your option, any later version of Perl 5 you may have available.