Web::Scraper - Web Scraping Toolkit inspired by Scrapi
use URI; use Web::Scraper; my $tweets = scraper { process "li.status", "tweets[]" => scraper { process ".entry-content", body => 'TEXT'; process ".entry-date", when => 'TEXT'; process 'a[rel="bookmark"]', link => '@href'; }; }; my $res = $tweets->scrape( URI->new("http://twitter.com/miyagawa") ); for my $tweet (@{$res->{tweets}}) { print "$tweet->{body} $tweet->{when} (link: $tweet->{link})\n"; }
Web::Scraper is a web scraper toolkit, inspired by Ruby's equivalent Scrapi. It allows you to write a web scraping script or class in a DSL-ish but still pure-perl language.
$scraper = scraper { ... };
Creates a new Web::Scraper object by wrapping the DSL code that will be fired when scrape method is called.
$res = $scraper->scrape(URI->new($uri)); $res = $scraper->scrape($html_content); $res = $scraper->scrape(\$html_content); $res = $scraper->scrape($http_response); $res = $scraper->scrape($html_element);
Retrieves the HTML from URI, HTTP::Response, HTML::Tree or text strings and creates a DOM object, then fires the callback scraper code to retrieve the data structure.
If you pass URI or HTTP::Response object, Web::Scraper will automatically guesses the encoding of the content by looking at Content-Type headers and META tags. Otherwise you need to decode the HTML to Unicode before passing it to scrape method.
You can optionally pass the base URL when you pass the HTML content as a string instead of URI or HTTP::Response.
$res = $scraper->scrape($html_content, "http://example.com/foo");
This way Web::Scraper can resolve the relative links found in the document.
scraper { process "tag.class", key => 'TEXT'; process '//tag[contains(@foo, "bar")]', key2 => '@attr'; };
process is the method to find matching elements from HTML with CSS selector or XPath expression, then extract text or attributes into the result stash.
If the first argument begins with "//" or "id(" it's treated as an XPath expression and otherwise CSS selector.
# <span class="date">2008/12/21</span> # date => "2008/12/21" process ".date", date => 'TEXT'; # <div class="body"><a href="http://example.com/">foo</a></div> # link => URI->new("http://example.com/") process ".body > a", link => '@href'; # <div class="body"><a href="http://example.com/">foo</a></div> # link => URI->new("http://example.com/"), text => "foo" process ".body > a", link => '@href', text => 'TEXT'; # <ul><li>foo</li><li>bar</li></ul> # list => [ "foo", "bar" ] process "li", "list[]" => "TEXT"; # <ul><li id="1">foo</li><li id="2">bar</li></ul> # list => [ { id => "1", text => "foo" }, { id => "2", text => "bar" } ]; process "li", "list[]" => { id => '@id', text => "TEXT" };
TBD
Tatsuhiko Miyagawa <miyagawa@bulknews.net>
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
http://blog.labnotes.org/category/scrapi/
To install Web::Scraper, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Web::Scraper
CPAN shell
perl -MCPAN -e shell install Web::Scraper
For more information on module installation, please visit the detailed CPAN module installation guide.