Extract cookies from open Safari session in Python

Question

Trying to follow the example in this one:

I'm trying to get files from this website and it's a tedious process of clicking to get to each file. I could easily use urllib2 to webscrape but, of course, you have to be logged in to this website to get the data. I tried doing one of the login type approaches in Python but I can't get it to work.

My other option is to export the cookies from Safari session so I can run my webscraping scripts to get the data I need.

Does anyone know how to export cookies from Safari session to Python for access to website data?

Use https://www.getpostman.com/ to see cookies and all the calls you make, but you need Chrome to use it — heinst, Jul 30 '15 at 17:23
Also why not use requests Session object to have a cookie jar and use their HTTPAuth methods as well?? — heinst, Jul 30 '15 at 17:23
@heinst hey thanks! can you link me to an example of a way to use requests session objects? — O.rka, Jul 30 '15 at 17:30
http://docs.python-requests.org/en/latest/user/advanced/#session-objects — heinst, Jul 30 '15 at 17:31
Whats the url for the site where you want to download the files from? — heinst, Jul 30 '15 at 17:41

heinst · Accepted Answer · 2015-07-30T18:03:48.887

This will allow you to login and download a file, this uses BeautifulSoup and Requests:

#!/usr/bin/python
from bs4 import BeautifulSoup
import requests


def download_file(url):
    local_filename = url.split('/')[-1]
    # NOTE the stream=True parameter
    r = requests.get(url, stream=True)
    with open(local_filename, 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024):
            if chunk: # filter out keep-alive new chunks
                f.write(chunk)
                f.flush()
    return local_filename

s = requests.Session()

soup = BeautifulSoup(s.get("https://minecraft.net/login").content)

authToken = soup.find('form', attrs={'id': 'loginForm'}).findNext('input')['value']

data = {
    "username": "USERNAME",
    "password": "PASSWORD",
    "remember": "true",
    "authenticityToken": authToken
}

s.post("https://minecraft.net/login", data=data)
soup = BeautifulSoup(s.get("https://minecraft.net//download").content)
download_file(soup.find("div", attrs={'id': 'platform-osx'}).findNext('p').findNext('')['href'])

Extract cookies from open Safari session in Python

1 Answers1