NAME

Scrappy - All Powerful Web Spidering, Scrapering, Crawling Framework

VERSION

version 0.9111110

SYNOPSIS

    #!/usr/bin/perl
    use Scrappy;

    my  $scraper = Scrappy->new;
        $scraper->crawl('search.cpan.org',
            '/recent' => {
                '#cpansearch li a' => sub {
                    print $_[1]->{href}, "\n";
                }
            }
        );

DESCRIPTION

Scrappy is an easy (and hopefully fun) way of scraping, spidering, and/or harvesting information from web pages, web services, and more. Scrappy is a feature rich, flexible, intelligent web automation tool.

Scrappy (pronounced Scrap+Pee) == 'Scraper Happy' or 'Happy Scraper'; If you like you may call it Scrapy (pronounced Scrape+Pee) although Python has a web scraping framework by that name and this module is not a port of that one.

METHODS

crawl

The crawl method is very useful when it is desired to crawl an entire website or at-least partially, it automates the tasks of creating a queue, fetching and parsing html pages, and establishing simple flow-control. See the SYNOPSIS for a simplified example, ... the following is a more complex example.

    my  $scrappy = Scrappy->new;
    
    $scrappy->crawl('http://search.cpan.org/recent',
        '/recent' => {
            
            '#cpansearch li a' => sub {
                my ($self, $item) = @_;
                # follow all recent modules from search.cpan.org
                $self->queue->add($item->{href});
            }
            
        },
        '/~:author/:name-:version/' => {
            
            'body' => sub {
                my ($self, $item, $args) = @_;
                
                my $reviews = $self
                ->select('.box table tr')->focus(3)->select('td.cell small a')
                ->data->[0]->{text};
                
                $reviews = $reviews =~ /\d+ Reviews/ ?
                    $reviews : '0 reviews';
                
                print "found $args->{name} version $args->{version} ".
                    "[$reviews] by $args->{author}\n";
                
            }
            
        }
    );

AUTHOR

Al Newkirk <awncorp@cpan.org>

COPYRIGHT AND LICENSE

This software is copyright (c) 2010 by awncorp.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.