Python mechanize wait and click

Question

Using mechanize, how can I wait for some time after page load (some websites have a timer before links appear, like in download pages), and after the links have been loaded, click on a specific link? Since it's an anchor tag and not a submit button, will browser.submit() work(I got errors while doing that)?

Answer 1

Mechanize does not offer javascript functionality, so you will not see dynamic content (like a timer that turns into a link).

As far as clicking on a link, you have to find the element, and then you can call click_link on it. See the Finding Links section of this site.

If you are looking for something to handle such sites, a good option is PhantomJS. It uses nodejs, but runs on the webkit engine, allowing you parse dynamic content. If you have your heart set on python, using Selenium to programatically drive a real browser may be your best bet.

Answer 2

If it's an anchor tag, then just GET/POST whatever it is.

The timer between links appearing is generally done in javascript - some sites you are attempting to scrape may not be usable without javascript, or requires a token generated in javascript with clientside math.

Depending on the site, you can either extract the wait time in msec/sec and time.sleep() for that long, or you'll have to use something that can execute javascript