Python: When using Requests package to perform a get request on a HTTPS page, how do I ensure that the sign in credentials are encripted

Question

So I'm using BeautifulSoup and Requests to get data from an https website

I know I need to provide authorisation details (I don't want to say verify=False)

Something like the solution in: How do I scrape an https page?

How do I ensure that the login and password are protected and not sent as plain text (I'm a rookie)

Is there anything I need to consider here?

a get request doesnt have a payload/body, only headers and url parameters, so they can be scraped from the url — mast3rd3mon, Jul 10 '18 at 11:52

score 0 · Accepted Answer · answered Jul 10 '18 at 12:11

I think you misunderstand the underlying technology concepts with regards to HTTP/S.

When you use BeautifulSoup, you Request a web page, and it returns the HTML. Some websites require you to authenticate to ensure content that is private remains protected.

A better question for you to ask would be:

"How do I use BeautifulSoup and Requests to scrape data from a website that requires authentication?"

And to that, I would answer, read the docs. Here is a link to get you started: Requests Sessions - http://docs.python-requests.org/en/v2.3.0/user/advanced/#session-objects

Python: When using Requests package to perform a get request on a HTTPS page, how do I ensure that the sign in credentials are encripted

1 Answers1