Jsoup login to scrape game data

Question

The question is can I use Jsoup to post login data that is controlled by javascript? Here's the info so far

Login URL for the site:

http://www.cybernations.net/login.asp

(they do have a no-bots policy, but I emailed the admin and have permission to auto-login for downloading game datafiles)

URL where files are stored

http://www.cybernations.net/stats_downloads.asp

The line of code where I use Jsoup to parse the html of the login page to show me the scripts...

Elements scriptTags = doc.getElementsByTag("script");

The output of looping through the list of Elements...

    <!--
function FrontPage_Form1_Validator(theForm)
{

  if (theForm.Username.value == "")
  {
    alert("Please enter a value for the \"Username\" field.");
    theForm.Username.focus();
    return (false);
  }

  if (theForm.Username.value.length > 40)
  {
    alert("Please enter at most 40 characters in the \"Username\" field.");
    theForm.Username.focus();
    return (false);
  }

  if (theForm.Validate_Password.value == "")
  {
    alert("Please enter a value for the \"Password\" field.");
    theForm.Validate_Password.focus();
    return (false);
  }

  if (theForm.Validate_Password.value.length < 1)
  {
    alert("Please enter at least 1 characters in the \"Password\" field.");
    theForm.Validate_Password.focus();
    return (false);
  }

  if (theForm.Validate_Password.value.length > 50)
  {
    alert("Please enter at most 50 characters in the \"Password\" field.");
    theForm.Validate_Password.focus();
    return (false);
  }
  return (true);
}
//-->

EDIT 1: edited the connection code Current code for login looks like so, returning the login page.

Connection.Response loginForm = Jsoup.connect( loginURL )
                        .method(Connection.Method.GET)
                        .execute();

Document document = Jsoup.connect( loginURL )
.data("Login", "Login")
.data("Username", user )
.data("Validate_Password", pass )
.cookies(loginForm.cookies() )
.post();

I feel like I'm missing something really simple here, should I direct the connect() method to follow redirects?

EDIT 2: Thanks for all your help, I think I'm going to switch to Apache's http client as it will (hopefully) give me greater control over the connection. Thank you all!

No exception, it just returned the html from the invalid login page. — Kyte, May 25 '14 at 02:23
FYI it’s __scrape__ (and __scraping__, __scraper__, __scraped__) not scrap. ‘To scrap’ means to throw away like rubbish :-( — DisappointedByUnaccountableMod, May 18 '21 at 18:49

score 1 · Answer 1 · edited Jun 20 '20 at 09:12

That function you posted is just there to validate the input, and you can ignore it, since the server probably doesn't allow usernames and passwords that don't meet their criteria anyway.

If you want to send the login information like the webpage does, you just need to POST to "/login.asp". Just look at the form in their HTML:

<form action="/login.asp" method="POST" name="FrontPage_Form1" .....

You'll have to handle the login yourself. You may need to read the cookies from the response header and remember them somewhere and then send them back with each subsequent request you make to the server (exactly as a web browser does it). Have a look at this for more information about that.

Also, you may need to consider how to handle captchas. It seems that their site forces you to pass a captcha after visiting the page twice, which will block your program from being able to log in.

Edit:

You can look at this answer for further information on how to automate the login. To answer your question about saving the cookies, it doesn't really matter where you save them, as long as you can access them when making additional requests to the server. That answer I just linked has code to access the cookies returned from the server when you log in (modified with your variables):

Connection.Response res = Jsoup.connect("http://www.cybernations.net/login.asp")
    .data("Username", "myUsername", "Validate_Password", "myPassword")
    .method(Method.POST)
    .execute();

Document doc = res.parse();
String sessionId = res.cookie("ASPSESSIONIDAAACSTQB");

That same answer shows you how to use jsoup to send the cookie in subsequent requests:

Document doc2 = Jsoup.connect("http://www.cybernations.net/stats_downloads.asp")
    .cookie("ASPSESSIONIDAAACSTQB", sessionId)
    .get();

Now, what the cookies you need to save exactly is something you need to figure out. Try using the developer options in Google Chrome. Log into the site, and see the names of the cookies the site is using to store your session (there are a few). Then try to emulate this with the above code.

I should mention that I have not tested this code for this site. That is something that will take time and patience, but that's part of the job.

So using Jsoup, what would the code to login look like? I've tinkered with a few different examples and I get returned to the login page (no captcha). — Kyte, May 24 '14 at 04:31
POST the username and password to the "action" link, like in [this question](http://stackoverflow.com/a/6714745/377628). Then there should be some cookies being set in the header, like `ASPSESSIONIDAAACSTQB`. Save those, and pass them in as headers when you make more requests to the server. Those cookies are how the server knows that you're logged in. — , May 24 '14 at 06:08
How would I save them? Should I let Jsoup save them for me or should I specify them manually? If you could update your answer with working example code, that would be great — Kyte, May 25 '14 at 02:24
@Kyte I added some specifics about using jsoup, but it's not a "working example". It is however, a good start to hopefully get you on the right track. — , May 25 '14 at 04:41

score 1 · Answer 2 · answered May 23 '14 at 22:56

form HTML element is the most important. You must check what is form method and the name of parameters.

<form action="/login.asp" method="POST" name="FrontPage_Form1" onsubmit="return FrontPage_Form1_Validator(this)" language="JavaScript" >
...
   <input value="" name="Username" id="Username" type="text" class="displayFieldIE" size="30" maxlength="40">
...
   <input value="" name="Validate_Password" id="Validate_Password" type="password" class="displayFieldIE" size="30" maxlength="50">
...
</form>

So you must post data to login.asp with parameters Username and Validate_Password.

Javascript you linked is here to validate user input. No need to deal with that.

score 1 · Answer 3 · answered May 25 '14 at 11:05

I don't see any problem in your approach. May be the site is checking for source. Try setting the referrer as

String loginURL = "http://www.cybernations.net/login.asp";
Connection.Response loginForm = Jsoup.connect(loginURL)
        .method(Connection.Method.GET).execute();

Document document = Jsoup.connect(loginURL)
            .data("Login", "Login")
            .data("Username", user)
            .data("Validate_Password", pass)
            .header("Host", "www.cybernations.net")
            .header("Origin", "http://www.cybernations.net")
            .referrer(loginURL)
            .cookies(loginForm.cookies())
            .post();

After first failed attempt, the site uses captcha. So be sure to pass correct credentials. ;)

If that didn't work try connecting via apache http client and pass the response to jsoup for parsing

I tried your code, and several different variants of it but no joy. Thanks for posting though! — Kyte, May 25 '14 at 16:43

Jsoup login to scrape game data

3 Answers3

Edit: