Name

WWW::PDAScraper - Class for scraping PDA-friendly content from websites

Synopsis

  use WWW::PDAScraper;
  my $scraper = WWW::PDAScraper->new qw ( NewScientist Yahoo::Entertainment );
  $scraper->scrape();
  

or

  use WWW::PDAScraper;
  my $scraper = WWW::PDAScraper->new;
  $scraper->scrape qw( NewScientist Yahoo::Entertainment );

or

  perl -MWWW::PDAScraper -e "scrape qw( NewScientist Yahoo::Entertainment )"

Description

Having written various kludgey scripts to download PDA-friendly content from various websites, I decided to try and write a generalised solution which would

* parse out the section of a news page which contains the links we want

* munge those links into the URL for the print-friendly version, if possible

* download those pages and make an index page for them

The moving of the pages to your PDA is not part of the scope of the module: the open-source browser and "distiller", Plucker, from http://plkr.org/ is recommended. Just get it to read the index.html file with a depth of 1 from disk, using a URL like file:///path/to/index.html

The Sub-modules

WWW::PDAScraper uses a set of rules for scraping a particular website from a second module, i.e. WWW::PDAScraper::Yahoo::Entertainment::TV contains the rules for scraping the Yahoo TV News website:

    package WWW::PDAScraper::Yahoo::Entertainment::TV;
    # WWW::PDAScraper.pm rules for scraping the
    # Yahoo TV website
    sub config {
        return {
            name       => 'Yahoo TV',
            start_from => 'http://news.yahoo.com/i/763',
            chunk_spec => [ "_tag", "div", "id", "indexstories" ],
            url_regex => [ '$', '&printer=1' ]
        };
    }
    1;

A more or less random selection of modules is included, as well as a full set for Yahoo, to demonstrate a logical set of modules in categories.

Creating a new sub-module ought to be relatively simple, see the template provided, WWW::PDAScraper::Template.pm - you need name, start_from, then either chunk_spec or url_spec, then optionally a url_regex for transformation into the print-friendly URL.

Then either move your new module to the same location as the other ones on your system, or make sure they're available to your script with a line like use lib '/path/to/local/modules/PDAScraper/'

USAGE

WWW::PDAScraper ought to be very simple to run, assuming you have the right sub-module(s).

It only has two main methods, new() and scrape(), and two supplementary ones, for assigning a proxy server to the user-agent and one for over-riding the default download location.

Either object-oriented, loading the sub-module(s) as part of "new":

  use WWW::PDAScraper;
  my $scraper = WWW::PDAScraper->new qw ( NewScientist Yahoo::Entertainment );
  $scraper->scrape();

or object-oriented, loading the sub-module(s) as part of each call to scrape():

  use WWW::PDAScraper;
  my $scraper = WWW::PDAScraper->new;
  $scraper->scrape qw( NewScientist Yahoo::Entertainment );
  $scraper->scrape qw( SomethingElse );

or procedural:

  use WWW::PDAScraper;
  scrape qw( NewScientist Yahoo::Entertainment );

or from the command line:

  perl -MWWW::PDAScraper -e "scrape qw( NewScientist Yahoo::Entertainment )"
  

The only extras involved would be adding a proxy to the user-agent and/or over-riding the default download location of $ENV{'HOME'}/scrape/

Object-oriented:

  use WWW::PDAScraper;
  my $scraper = WWW::PDAScraper->new;
  $scraper->proxy('http://your.proxy.server:port/');
  $scraper->download_location("/path/to/folder/");

procedural:

  use WWW::PDAScraper;
  proxy('http://your.proxy.server:port/');
  download_location("/path/to/folder/");

I wish I didn't need this code

In the days of modern web publishing, I shouldn't need to create this code. All websites should make themselves PDA-friendly by the use of client detection or smart CSS or XML. But they don't.

Bugs

The websites will certainly change, and at that time the sub-modules will stop working. There's no way around that.

Obviously it would be useful if there were a developer/user community which contributed new modules and updated the old ones.

See Also

HTML::Element, for the syntax of chunk_spec in sub-modules.

To do

The user-agent should really be part of the object, I guess. That would be neater.

And it should actually use WWW::Robot instead of LWP so it doesn't hammer servers.

And we could either add arbitrary numbers of regexes for fixing up the pages of sites which don't have a print-friendly version of the page, or add a second level of parsing to find the print-friendly link, for sites which don't have a logical relationship between the regular link and the print-friendly.

Author

        John Horner
        CPAN ID: CODYP
        
        bounce@johnhorner.nu
        http://pdascraper.johnhorner.nu/

Copyright

This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

cpanm WWW::PDAScraper

perl -MCPAN -e shell
install WWW::PDAScraper