Alternative to using mechanize to fetch data?

Question

Currently, I am using mechanize to fill up a form and send a POST request, then I am doing a regex search on the response to get the data ( a floating point number).

Is there any way I can do this by just sending a POST request? I know this is possible by using a combination of any browser's Developer Tools and the requests module to send the request but I have failed to find a comprehensive tutorial. I would also like some details about the steps involved.

Answer 1

First step: get the field name

Inspect the HTML code and find the name attribute of the field. For example, the comment form on this page is (in Chrome, right-click and choose "inspect element"):

<textarea name="comment" cols="68" rows="3" 
placeholder="Use comments to ask for more information or 
suggest improvements. Avoid answering questions in comments."
></textarea>

The field name is comment.

Step 2: assemble a dict of name: value for each field (including the hidden inputs)

Lets call it data:

data = {
   "comment": "this is a comment",
   "post_id": 1234
}

Step 3: use the data argument of `requests.post'

response = requests.post(url, data=data, cookies=cookies)

More advanced stuff

If your form has file fields you may have to check "More complicated POST requests" in the docs. Same goes for custom authentication, cookie handling, etc.