NAME

WWW::Crawler::Mojo::UserAgent - Crawler specific featured user agent

SYNOPSIS

    my $ua = WWW::Crawler::Mojo::UserAgent->new;
    $ua->keep_credentials(1);
    $ua->credentials(
        'http://example.com:8080' => 'jamadam:password1',
        'http://example2.com:8080' => 'jamadam:password2',
    );
    my $tx = $ua->get('http://example.com/');
    say $tx->req->url # http://jamadam:passowrd@example.com/
    
    if ($ua->active_conn < $max_conn) {
        $ua->get(...);
    }
    
    if ($ua->active_host($url) < $max_conn_per_host) {
        $ua->get(...);
    }

DESCRIPTION

This class inherits Mojo::UserAgent and adds credential storage and active connection counter.

ATTRIBUTES

WWW::Crawler::Mojo::UserAgent inherits all attributes from Mojo::UserAgent.

active_conn

A number of current connections.

    $bot->active_conn($bot->active_conn + 1);
    say $bot->active_conn;

active_conn_per_host

A number of current connections per host.

    $bot->active_conn_per_host($bot->active_conn_per_host + 1);
    say $bot->active_conn_per_host;

keep_credentials

Sets true to activate the feature. Defaults to 1.

    $ua->keep_credentials(1);

METHODS

WWW::Crawler::Mojo::UserAgent inherits all methods from Mojo::UserAgent.

active_host

Maintenances the numbers of active connections.

    $ua->active_host($url, 1);
    $ua->active_host($url, -1);
    my $amount = $ua->active_host($url);

credentials

Stores credentials.

    $ua->credentials(
        'http://example.com:8080' => 'jamadam:password1',
        'http://example2.com:8080' => 'jamadam:password2',
    );

new

Constructer.

    $ua = WWW::Crawler::Mojo::UserAgent->new;

AUTHOR

Keita Sugama, <sugama@jamadam.com>

COPYRIGHT AND LICENSE

Copyright (C) Keita Sugama.

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

cpanm WWW::Crawler::Mojo

perl -MCPAN -e shell
install WWW::Crawler::Mojo