WWW::Mechanize::FAQ - Frequently Asked Questions about WWW::Mechanize
That's because WWW::Mechanize doesn't operate on the JavaScript. It only understands the HTML parts of the page.
It does pretty much, but it doesn't support JavaScript.
I added some basic attempts at picking up URLs in window.open() calls and return them in $mech->links. They work sometimes. Beyond that, there's no support for JavaScript.
window.open()
$mech->links
I will if anyone sends me the code to do it. I'm not going to write a JavaScript processor myself.
Yes.
Hard enough that I don't want to deal with it myself. Plus, I don't use JavaScript myself, so I don't have an itch to scratch.
I've heard noises from people every so often over the past couple of years, but nothing you'd pin your hopes on.
I'm sure it would.
I have no idea if or when such a thing will ever get done. I can guarantee that as soon as there's anything close to JavaScript support I will let everyone know.
If you must, but I doubt that anyone's written JavaScript support for Mechanize and neglected to tell me about it.
Since Javascript is completely visible to the client, it cannot be used to prevent a scraper from following links. But it can make life difficult, and until someone writes a Javascript interpreter for Perl or a Mechanize clone to control Firefox, there will be no general solution. But if you want to scrape specific pages, then a solution is always possible.
One typical use of Javascript is to perform argument checking before posting to the server. The URL you want is probably just buried in the Javascript function. Do a regular expression match on $mech->content() to find the link that you want and $mech->get it directly (this assumes that you know what your are looking for in advance).
$mech->content()
$mech->get
In more difficult cases, the Javascript is used for URL mangling to satisfy the needs of some middleware. In this case you need to figure out what the Javascript is doing (why are these URLs always really long?). There is probably some function with one or more arguments which calculates the new URL. Step one: using your favorite browser, get the before and after URLs and save them to files. Edit each file, converting the the argument separators ('?', '&' or ';') into newlines. Now it is easy to use diff or comm to find out what Javascript did to the URL. Step 2 - find the function call which created the URL - you will need to parse and interpret its argument list. Using the Javascript Debugger Extension for Firefox may help with the analysis. At this point, it is fairly trivial to write your own function which emulates the Javascript for the pages you want to process.
If it's possible with LWP::UserAgent, then yes. WWW::Mechanize is a subclass of LWP::UserAgent, so all the wondrous magic of that class is inherited.
See the docs in LWP::UserAgent on how to use the proxy. Short version:
$mech->proxy(['http', 'ftp'], 'http://proxy.example.com:8000/');
or get the specs from the environment:
$mech->env_proxy(); # Environment set like so: gopher_proxy=http://proxy.my.place/ wais_proxy=http://proxy.my.place/ no_proxy="localhost,my.domain" export gopher_proxy wais_proxy no_proxy
Use the mech-dump utility, optionaly installed with Mechanize.
$ mech-dump --forms http://search.cpan.org Dumping forms GET http://search.cpan.org/search query= mode=all (option) [*all|module|dist|author] <NONAME>=CPAN Search (submit)
use MIME::Base64; my $agent = WWW::Mechanize->new(); my @args = ( Authorization => "Basic " . MIME::Base64::encode( USER . ':' . PASS ) ); $agent->credentials( ADDRESS, REALM, USER, PASS ); $agent->get( URL, @args );
You can't. JavaScript is entirely client-based, and WWW::Mechanize is a client that doesn't understand JavaScript.
You don't deal with them as frames, per se, but as links. Extract them with
my @frame_links = $mech->find_link( tag => "frame" );
All HTTP::Headers methods work on a HTTP::Response object which is returned by the get(), reload(), response()/res(), click(), submit_form(), and request() methods.
my $mech = WWW::Mechanize->new( autocheck => 1 ); $mech->get( 'http://my.site.com' ); my $res = $mech->response(); for my $key ( $response->header_field_names() ) { print $key, " : ", $response->header( $key ), "\n"; }
Run mech-dump on your page and see what it says.
mech-dump is a marvelous diagnostic tool for figuring out what forms and fields are on the page. Say you're scraping CNN.com, you'd get this:
$ mech-dump http://www.cnn.com/ GET http://search.cnn.com/cnn/search source=cnn (hidden readonly) invocationType=search/top (hidden readonly) sites=web (radio) [*web/The Web ??|cnn/CNN.com ??] query= (text) <NONAME>=Search (submit) POST http://cgi.money.cnn.com/servlets/quote_redirect query= (text) <NONAME>=GET (submit) POST http://polls.cnn.com/poll poll_id=2112 (hidden readonly) question_1=<UNDEF> (radio) [1/Simplistic option|2/VIEW RESULTS] <NONAME>=VOTE (submit) GET http://search.cnn.com/cnn/search source=cnn (hidden readonly) invocationType=search/bottom (hidden readonly) sites=web (radio) [*web/??CNN.com|cnn/??] query= (text) <NONAME>=Search (submit)
Four forms, including the first one duplicated at the end. All the fields, all their defaults, lovingly generated by HTML::Form's dump method.
dump
If you want to run mech-dump on something that doesn't lend itself to a quick URL fetch, then use the save_content() method to write the HTML to a file, and run mech-dump on the file.
save_content()
You need either IO::Socket::SSL or Crypt::SSLeay installed.
You're trying to change the value of a hidden field and you have warnings on.
First, make sure that you actually mean to change the field that you're changing, and that you don't have a typo. Usually, hidden variables are set by the site you're working on for a reason. If you change the value, you might be breaking some functionality by faking it out.
If you really do want to change a hidden value, make the changes in a scope that has warnings turned off:
{ local $^W = 0; $agent->field( name => $value ); }
Are you checking your errors?
Are you sure?
Are you checking that your action succeeded after every action?
For example, if you try this:
$mech->get( "http://my.site.com" ); $mech->follow_link( "foo" );
and the get call fails for some reason, then the Mech internals will be unusable for the follow_link and you'll get a weird error. You must, after every action that GETs or POSTs a page, check that Mech succeeded, or all bets are off.
get
follow_link
$mech->get( "http://my.site.com" ); die "Can't even get the home page: ", $mech->response->status_line unless $mech->success; $mech->follow_link( "foo" ); die "Foo link failed: ", $mech->response->status_line unless $mech->success;
$mech->get($url)
There are many reasons why a get() can fail. The server can take you to someplace you didn't expect. It can generate redirects which are not properly handled. You can get time-outs. Servers are down more often than you think! etc, etc, etc. A couple of places to start:
get()
$mech->uri()
LWP::Debug
If things are really strange, turn on debugging with use LWP::Debug qw(+); Just put this in the main program. This causes LWP to print out a trace of the HTTP traffic between client and server and can be used to figure out what is happening at the protocol level.
use LWP::Debug qw(+);
It is also useful to set many traps to verify that processing is proceeding as expected. A Mech program should always have an "I didn't expect to get here" or "I don't recognize the page that I am processing" case and bail out.
Since errors can be transient, by the time you notice that the error has occurred, it might not be possible to reproduce it manually. So for automated processing it is useful to email yourself the following information:
where processing is taking place
An Error Message
$mech->uri
$mech->content
You can also save the content of the page with $mech->save_content( 'filename.html' );
$mech->save_content( 'filename.html' );
The post is handled by application software. It is common for PHP programmers to use the same file both to display a form and to process the arguments returned. So the first task of the application programmer is to decide whether there are arguments to processes. The program can check whether a particular parameter has been set, whether a hidden parameter has been set, or whether the submit button has been clicked. (There are probably other ways that I haven't thought of).
In any case, if your form is not setting the parameter (e.g. the submit button) which the web application is keying on (and as an outsider there is no way to know what it is keying on), it will not notice that the form has been submitted. Try using $mech->click() instead of $mech->submit() or vice-versa.
$mech->click()
$mech->submit()
Some web sites use distributed databases for their processing. It can take a few seconds for the login/session information to percolate through to all the servers. For human users with their slow reaction times, this is not a problem, but a Perl script can outrun the server. So try adding a sleep(5) between logging in and actually doing anything (the optimal delay must be determined experimentally).
Mech keeps a history of every page, and the state it was in. It actually keeps a clone of the full Mech object at every step along the way.
You can limit this stack size with the stack_depth parm in the new() constructor.
stack_depth
new()
Copyright 2005 Andy Lester <andy at petdance.com>
<andy at petdance.com>
To install WWW::Mechanize, copy and paste the appropriate command in to your terminal.
cpanm
cpanm WWW::Mechanize
CPAN shell
perl -MCPAN -e shell install WWW::Mechanize
For more information on module installation, please visit the detailed CPAN module installation guide.