Jsoup login problem (URL/cookie/session/POST/data)

Question

My goal is to login to this website.

After crawling through various threads:
- jsoup posting and cookie
- Login to Facebook with Jsoup and proper cookies
- How to post form login using jsoup?
- Cannot login to website using jsoup

I finally came up with this test class:

public class JsoupTest {
    public static void main(String args[]) throws URISyntaxException {
        try {

            String urlLogIn = "https://invest.firstrade.com/cgi-bin/login";

            // Put the url that you see when you have logged in.
            String urlUnderTest = "https://invest.firstrade.com/cgi-bin/main#/cgi-bin/acctpositions"; 

            // lets make data map containing all the parameters and its values found in the form
            Map<String, String> mapParams = new HashMap<String, String>();
            mapParams.put("redirect", "");
            mapParams.put("ft_locale", "en-us");
            mapParams.put("login.x", "Log In");
            mapParams.put("username", "MY_USERNAME");
            mapParams.put("password", "MY_PASSWORD");
            mapParams.put("destination_page", "acctpositions");

            print("started");

            // With this you login and a session is created
            Connection.Response res = Jsoup.connect(urlLogIn)
                    .data(mapParams)
                    .method(Method.POST)
                    .execute();

            // This will get you cookies
            Map<String, String> loginCookies = res.cookies();

            // Here you parse the page that you want.
            Document doc = Jsoup.connect(urlUnderTest).cookies(loginCookies).get();

            System.out.println(doc.title());
            print(doc.toString());

            print("done");

        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    private static void print(String msg, Object... args) {
        System.out.println(String.format(msg, args));
    }
}

Sadly, no matter what modification I make, I get stuck on a page saying "session failed".

<html>
 <head>
  <script>window.location = "/cgi-bin/sessionfailed?reason=6"</script>
 </head>
 <body>
  Please login first.
  <br>
  <br>
  <a href="/cgi-bin/login">Go to Login Page</a>
  <br>
 </body>
</html>

However I can use this class to login facebook successfully, using the URL provided in this thread:
- https://stackoverflow.com/a/49984544/10857019

Jsoup.connect("https://m.facebook.com/login/async/?refsrc=https%3A%2F%2Fm.facebook.com%2F&lwv=100")

So I'm quite confused whether the problem is in URLs or cookie/session or sth else?

Any help would be greatly appreciated. Thanks so much for reading this thread!

Can't be reproduced without having an account. In general: open dev tools in browser and monitor the requests for the login in the network tab, then replicate the requests with all headers (e.g. set User-Agent, etc.), there might be interim cookies, js based modifications of the cookies/challenges (disable js, clear cookies and try to login manually), etc. — Frederic Klein, Jan 03 '19 at 09:25

score 0 · Accepted Answer · answered Jan 11 '19 at 23:52

Thanks for Frederic's comment.

I figured out the answer myself.
The point I was missing for this case is the persisting cookies.
In the below solution I

Create a map (call it session) to store this data

Update it every time I've sent a request, using putAll method
session.putAll(resp.cookies());

Include this map in the next request
.cookies(session)

Note that 2 and 3 must be repeated in every request thereafter.
Basically that's all!

public class JsoupTest {
    public static void main(String args[]) throws URISyntaxException {
        try {

            String urlLogIn = "https://invest.firstrade.com/cgi-bin/login";

            // Put the url that you see when you have logged in.
            String urlUnderTest = "https://invest.firstrade.com/cgi-bin/main#/cgi-bin/acctpositions"; 

            // Create a "session" map here to persists cookies across all requests
            Map<String, String> session = new HashMap<String, String>();

            // get initial login form
            Response loginForm = Jsoup.connect(urlLogIn)
                    .userAgent(agent)
                    .method(Method.GET)
                    .execute();

            // initialize session
            session = loginForm.cookies();



            print("started");



            // data map containing all the parameters and its values found in the form
            Map<String, String> mapParams = new HashMap<String, String>();
            mapParams.put("redirect", "");
            mapParams.put("ft_locale", "en-us");
            mapParams.put("login.x", "Log In");
            mapParams.put("username", "MY_USERNAME");
            mapParams.put("password", "MY_PASSWORD");
            mapParams.put("destination_page", "acctpositions");

            // With this you login and a session is created
            Connection.Response res = Jsoup.connect(urlLogIn)
                    .userAgent(agent)
                    .data(mapParams)
                    .cookies(session)
                    .method(Method.POST)
                    .execute();

            // update session after login
            session.putAll(loginResp.cookies());



            print("done");



            // The snippet from "started" to "done" is a full cycle
            // From here on every request is basically as the snippet above
            // For example:
            mapParams = new HashMap<String, String>();
            mapParams.put("other required form data", "DATA");

            resp = Jsoup.connect(urlUnderTest)
                    .userAgent(agent)
                    .data(mapParams)
                    .referrer("https://invest.firstrade.com/cgi-bin/main") // sometimes referrer is necessary
                    .cookies(session)
                    .method(Method.POST)
                    .execute();
            session.putAll(resp.cookies());

            // Then you start your crawler stuff
            Document doc = resp.parse();
            System.out.println(doc.title());
            print(doc.toString());

        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    private static void print(String msg, Object... args) {
        System.out.println(String.format(msg, args));
    }
}

Jsoup login problem (URL/cookie/session/POST/data)

1 Answers1