0

I need to scrape private data from a Japanese site. But I can't login into the site and receive error code 422. How can I login?

I need to log in into this website: https://moneyforward.com/users/sign_in

The form on this site is:

<form class="form-horizontal mf-mb40" id="new_sign_in_session_service" action="/session" accept-charset="UTF-8" method="post">
<input name="utf8" type="hidden" value="&#x2713;" />
<input type="hidden" name="authenticity_token" value="OGuijdFq6M1xngenCHi0BgZh9x0Nniw2HxiRhC9H2T0vbgWcWNRz+fmi5wxEdk4ua5TL9/UF7BapR2Af8CdILQ==" />
<div class="a-text--f14 mb3">e-mail</div><div class="a-text-box">
<input placeholder="entry" class="js-focus-form" type="email" name="sign_in_session_service[email]" id="sign_in_session_service_email" />
</div>
<div class="a-text--f14 mb3 mt20">password</div><div class="a-text-box">
<input placeholder="entry" type="password" name="sign_in_session_service[password]" id="sign_in_session_service_password" />
</div>
<div class="m-password-support">
<div class="a-checkbox password-show">
<label><input class="checkbox" id="show-ps" name="new1" type="checkbox" value="1" />
<span class="checkbox-icon"></span>
<span class="checkbox-txt">password</span></label>
</div>
<div class="password-forget">
<a href="/password_reset/new">forgot password</a>
</div>
</div>
<div class="a-button primary">
<input type="submit" name="commit" value="login" id="login-btn-sumit" data-disable-with="login" />
</div>
</form>

I receive the error on last row "var response = UrlFetchApp.fetch(url, options)" below code.

function Login() {

    var account ='***';
    var password = '***';

    var response = UrlFetchApp.fetch("https://moneyforward.com/users/sign_in");

    var regexp     = /<input type=\"hidden\" name=\"authenticity_token\" value=\"(.*?)\" \/>/;
    var elements = response.getContentText().match(regexp);

    var headers = response.getAllHeaders();
    var cookies = [];
    if ( typeof headers['Set-Cookie'] !== 'undefined' ) {
        var cookies = typeof headers['Set-Cookie'] == 'string' ? [ headers['Set-Cookie'] ] : headers['Set-Cookie'];
        for (var i = 0; i < cookies.length; i++) {
            cookies[i] = cookies[i].split( ';' )[0];
        };
    };

    var payload = {
        utf8: "✓",
        authenticity_token : elements[1],
        email : "account",
        password : password
    };

    var headers = { 'Cookie' : cookies };

    options = {
        method : "post",
        headers : headers,
        followRedirects: true, 
        contentType: "application/x-www-form-urlencoded",
        //muteHttpExceptions : true,
        payload : payload,
    };

    var url = "https://moneyforward.com/session";
    var response = UrlFetchApp.fetch(url, options);

}
mar
  • 1
  • 1
  • Why have you got two sets of instructions for `UrlFetchApp` plus different URLs in the same script? Looks like you have combined [Google Apps Script login to website with HTTP request](https://stackoverflow.com/questions/21621019/) and [Google Script authenticate on external website](https://stackoverflow.com/questions/37675406/), but you're having two-bob each way by combining them all into one script. Just try one at a time and see what success/error you get. – Tedinoz Jan 27 '19 at 11:19
  • Thanks! The reason of two sets: the first set is for getting authenticity_token value and cookies, second set is to login into a page that I want to get data. "Authenticity_token" value and cookies are updated at every my accesing to the page, so I need to get the sets. I think I have to have this proceed in order to login with HTTP request and call to the same website with different URL. I am sorry I can't know what do you mean. I'm a begginer at google apps script. Do you mean that this function should be separated "Getting data function" and "Login function"? – mar Jan 30 '19 at 02:24
  • Two things. One: Please provide the _full and complete_ error message that you get when you login. Two: You ask "Do you mean that this function should be separated "Getting data function" and "Login function"? Yes; that exactly what I mean. The problem that you asked was "how do I login to this site"? But the "login" function handles the initial login **as well as** the scraping of your data (which is why it contains 2 x `UrlFetchApp`). But if you can't login then the rest is a distraction. So, work on building a "login" function that works; then you can add the code for scraping your data. – Tedinoz Jan 30 '19 at 03:22

0 Answers0