7

I'm using Python 3 for a script that will monitor updates in a user's profile on a webpage. The login to this site is protected by CSRF countermeasures, which is a good thing. However, I can't get my script to login to this site.

  • My approach using mechanicalsoup:

    import mechanicalsoup

browser = mechanicalsoup.Browser()
login_page = browser.get(base_url)
login_form = login_page.soup.select(".form-signin")[0]

login_form.find(attrs={"name": "username"})['value'] = 'username'
login_form.find(attrs={"name": "password"})['value'] = 'password'

page2 = browser.submit(login_form, login_url)
print(str(page2.text))

  • My approach using robobrowser:

    import re
from robobrowser import RoboBrowser

browser = RoboBrowser(history=True)
browser.open(base_url)
form = browser.get_form(action='/login/')

form["username"] = 'username'
form["password"] = 'password'

browser.submit_form(form)
print(str(browser.select))

In both cases I end up with a HTTP status of 403 and a message saying CSRF verification failed. Request aborted.

  • Any ideas how to fix this?
  • The form in question has a hidden input containing a CSRF token. I guess mechanicalsoup and robobrowser will submit this input as well. Am I right? Or do I have to treat it specially?
  • I thought the session used by this two packages would handle everything like cookies and so on. Is there something I've missed?
Scolytus
  • 16,338
  • 6
  • 46
  • 69

2 Answers2

10

I got the robobrowser variant to work by setting the Referer header.

browser.session.headers['Referer'] = base_url

So the complete code that worked for me is the following:

import re
from robobrowser import RoboBrowser

browser = RoboBrowser(history=True)
browser.open(base_url)
form = browser.get_form(action='/login/')

form["username"] = 'username'
form["password"] = 'password'
browser.session.headers['Referer'] = base_url

browser.submit_form(form)
print(str(browser.select))
Scolytus
  • 16,338
  • 6
  • 46
  • 69
1

You are only adding username and password to the form you are submitting, you need to add the csrf token field as well. See below, I'm assuming you can figure out the field name and token value.

form["username"] = 'username'
form["password"] = 'password'
form["csrffieldname"] = 'csrfvalue' # This is what you are missing

The token value will be different for each form submission, so you'll have to get the form and parse out the csrf token value and submit it before the timeout expires for the token.

mikeb
  • 10,578
  • 7
  • 62
  • 120
  • The csrf is already in the form. Manually adding it to the form again doesn't change the behavior. Still 403. – Scolytus Jul 24 '15 at 11:16
  • Can you iterate over and print the values in form right before this: `browser.submit_form(form)`? – mikeb Jul 24 '15 at 11:17
  • Thanks for the effort, but the problem was the missing `Referer` header. I checked the form object as you suggested, but it had everything in it. It looks like this is a special behavior of this particular web app, since I can't see why the referrer would be necessary. – Scolytus Jul 27 '15 at 19:59