2

I'm writing a google apps script that is supposed to login to a website and then crawl the site and perform different actions.

The script successfully authenticates the website and completes several get methods, but when it tries to do a Post on a PostSelect drop-down box I'm getting a 500 response code.

My script follows:

    function getRaceData(browser) {
      var eId = "72370";
      var selectNbr = "17";
      var url = "http://tnetwork.trakus.com/tnet/t_Recap.aspx?EventID=" + eId;
      var options = {
               "method": "get",
                // Set the cookies so that we appear logged-in
                "headers": {
                            "Cookie": browser.cookie
                           },
                "followRedirects" : false
              }

     var response = UrlFetchApp.fetch(url, options);
     var html = response.getContentText();
     var code = response.getResponseCode();
     browser.html = html;
     Utilities.sleep(1000);
     var url = "http://tnetwork.trakus.com/tnet/t_Recap.aspx";
     var payload = {"EventIDHidden" : eId, "PostSelect" : selectNbr};
     var header = {"Cookie" : browser.cookie};
     var options = {"method" : "post","headers" : header,"payload":payload,"followRedirects" : false};         

    Logger.log(UrlFetchApp.getRequest(url, options));

 **var response = UrlFetchApp.fetch(url, options);**      //This line causes 500 response code.
     var html = response.getContentText();
     var code = response.getResponseCode();
     Logger.log(html);
    }

Here is the error message I receive:

Request failed for http://tnetwork.trakus.com/tnet/t_Recap.aspx returned code 500. Truncated server response: <html>
 <head>
 <title>Runtime Error</title>
 <style>
 body {font-family:"Verdana";font-weight:normal;font-size: .7em;... (use muteHttpExceptions option to examine full response) (line 190, file "PostTest")

Update: Here is the Request Header that I was able to inspect by using Google Chrome Developer Tool Network selection:

POST /tnet/t_Recap.aspx HTTP/1.1
Host: tnetwork.trakus.com
Connection: keep-alive
Content-Length: 23079
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://tnetwork.trakus.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.76 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://tnetwork.trakus.com/tnet/t_Recap.aspx?EventID=72370
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: ASP.NET_SessionId=ayek3u55dsapvxe2t5kio0a3; __utma=260442568.815817400.1426544670.1426555405.1426631681.3; __utmc=260442568; __utmz=260442568.1426544670.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); userCredentials=username=MyUserName; .ASPXAUTH=D637CD472308F766FA8D267B85D8018028FC7981748D6B3EE9DC43166998E1F7F5A9B2A676CB1FD67560B86F40A633C4DDAAE6B13BBD0B07728E00E00977F271ABD5CC74D9AD88405CB7DD04F43F5EDDEF84EAAA14E5A3A638A744B0B76EA2FFEF2434A393E5FD699416FEA1A4EF0488671D82E4; __utma=190106350.902310224.1426544678.1426631684.1426643061.5; __utmb=190106350.3.10.1426643061; __utmc=190106350; __utmz=190106350.1426544678.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)

Also here is what I get when from logging the my UrlFetchApp.getRequest(url, options) function.

{headers{Cookie=ASP.NET_SessionId=zv2o2o55tgl2ctbse52vwln1;userCredentials=username=MyUserName;.ASPXAUTH=9980822DFE22641E55F3545CAEAFA748A39AC05A69F74A67BC656078C192769BB74A0B6F000241B36A670362019F5D14DCD1A6FD9E43718B5BC8AD89E9037DBEA1CE2A012AFFFC704D35C71EB82DAC18C644F0EBA34992604ABF2680DE97098C8BE3CB0BAEADD51C65E2EB0C8851C9E151337EA4, X-Forwarded-For=71.168.117.91}, useIntranet=false, followRedirects=false, payload=PostSelect=17&EventIDHidden=72370, method=post, validateHttpsCertificates=true, contentType=application/x-www-form-urlencoded, url=http://tnetwork.trakus.com/tnet/t_Recap.aspx}

I believe my syntax is correct and I've been able to pass the cookie successfully with all of the prior get statements and for the Login Post, so I'm not sure why this Post is failing. If anyone has any thoughts or has come across something similar, I'd appreciate your input

Update: I been review the page source a bit more carefully and wanted to share the following:

That page has a very large form that appears to include hidden inputs other than the ones in my Post statement (See below). Additionally, the Select statement has an onChange="UpdatePostView().

The UpdatePostView() is fairly straight forward:

function UpdatePostView()
    {
        document.TChartForm.submit();
    }

Additional inputs include

    <form name="TChartForm" method="post" action="t_Recap.aspx" id="TChartForm">
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="eBDduN+CtRzqAG6KZjuiZQhrTRokT2cqgaabq7fy3oSZaeE+O83K0QTLMCessWWC27snsJrG607bicynkgi9j1+4a++Gpz8+S5CeD9hhjzEEH8NuJUOtgPDOV5QFkqNrGVX4g+1YehwD+A5V7K5o+j5zva1cABIEhbHoxAqKrETpZgNVDjpZWTc+Hl4NaG1aTGpARvzUTgU23CvtxwYrIAH2CVuDHeqMTo0B9aOB73e4liFQtMnj/64ETSf2eGvbgmBLKaPHDxDrRCjMRLBinWRn94lLF6Cf3ZUVoQTMU4IgpQ==" />

        <input type="hidden" name="__VIEWSTATEGENERATOR" id="__VIEWSTATEGENERATOR" value="696C68DE" />
        <input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION"    value="pv3ODePltFyNx0vx4zFN/4siIO4psfHIMLofolP8T27GF4kko1Ol7wQaPQ0nuAtGCoANyMCeHmV0ecBuv/CGuv4YF2jZSIQmPtNFmSpDEh/EJvyVxljXY2d+gv8=" />     
                                <input type="hidden" name="EventIDHidden" id="EventIDHidden" value="72370" />
                                <input type="hidden" id="SessionUser" value="MyUserName" />
                                <input type="hidden" id="SessionAccessStr" value="MyUserName" />
                                <div id="DisplayArea"><table class="headLinks" align="center" width="97%" cellpaddiong="0" cellspacing="0"><tr><td width="50%"><div align="left"><a style="white-space:nowrap;" href="t_RaceDay.aspx?VenueID=24&amp;Type=TBRED&amp;Date=03/01/2015">&laquo;&laquo;&laquo;More Aqueduct Races on 03/01/2015</a></div><br /><div align="left"><a href="previous.html" onClick="history.back();return false;">&laquo;&laquo;&laquo;Previous Page
            </a></div><p /></td><td valign="top"><div align="right"><b>
              T-Net Member: myTnetMemberName </b></div></td></tr></table><table align="center" class="topThreeContainer" cellpadding="2" cellspacing="0"><tr align="center" class="topThreeTitle"><td style="white-space:nowrap">Track</td><td style="white-space:nowrap">Official Results</td><td style="white-space:nowrap">Trakus Times</td></tr><tr><td><img src="images/TrackLogos/24.gif" /></td><td valign="top"><table cellpadding="2"><tr class="topThreeHead"><td><div style="width:40px;">Win</div></td><td><div style="width:40px;">Place</div></td><td><div style="width:40px;">Show</div></td></tr><tr><td><img width="30" height="27" src="images/HorseRacingTiles/Trakus/3.gif" /></td><td><img width="30" height="27"  

additional form data omitted due to space constraints....

<select runat="server" autopostback="true" style="width:250px" id="PostSelect" onChange="UpdatePostView();" name="PostSelect"><option value="0">Race Summary</option><option value="2">1/16 Mile</option><option value="3">1/8 Mile</option><option value="4">3/16 Mile</option><option value="5">1/4 Mile (POC)</option><option value="6">5/16 Mile</option><option value="7">3/8 Mile</option><option value="8">7/16 Mile</option><option value="9">1/2 Mile (POC)</option><option value="10">9/16 Mile</option><option value="11">5/8 Mile</option><option value="12">11/16 Mile</option><option value="13">3/4 Mile (POC)</option><option value="14" selected="">13/16 Mile</option><option value="15">7/8 Mile (POC)</option><option value="16">15/16 Mile</option><option value="17">1 Mile (Finish)</option></select>

additional form data omitted due to space constraints....

 Race Notes: </div></td></tr><tr><td align="right" colspan="2"><div align="right" style="font-size:8px;font-weight:bold;">Updated 3/1/2015 6:29:10 PM GMT
</div></td></tr></table></div></form>

Given this new information, I have a couple additional questions:

  1. Do I need to add the other hidden inputs to my payload or are they embedded in the Html and therefore get submitted anyway.
  2. Does the urlfecthapp command fire the onchange event or do I have to do something special to fire that.
  3. What is the role of the ViewState, ViewStateGenerator and EventValidation. These were present in the login form, but I did not need to include them in the payload to login successfully.

Here is my full script if anyone would like to try it and see the results for themselves.

Full Script

Thanks again.

Mutuelinvestor
  • 3,384
  • 10
  • 44
  • 75
  • In your code sample, you mark that getRequest is returning the 500. That doesn't actual execute the request. Did you mean the following line? Also, are there any additional details about the 500 error? Anyway, chances are that if the 500 error is coming from trakus.com then you're doing something it doesn't expect. Without knowing a lot more about the app and what you're trying to do, it's hard to offer any specific advice. All I can suggest is make sure you run through the same flow interactively in your browser and compare requests you see there vs. what you're scripting. – Steve Bazyl Mar 20 '15 at 18:47
  • @SteveBazyl you are correct, I did mean the line below it. I've updated the question. I've spent the last week of evenings in Google's Developer tools going through each step in the chrome browser and comparing what I see in the Network section of the development tools to my code. This has me stumped, presently. Thank for the feedback and nice catch on the question. – Mutuelinvestor Mar 20 '15 at 19:27
  • @SteveBazyl if you have some time and want to take a look at the site, I'd be more than happy to share a pw and uid with you. As I said, I've been trying to get this one for a week. – Mutuelinvestor Mar 20 '15 at 19:31
  • Yes, you'll need to submit all the form parameters yourself. Your script has to emulate what the browser would normally do. As for what those extra parameters do, that's internal to the app. – Steve Bazyl Mar 23 '15 at 17:28
  • possible duplicate of [Why is my urlFetchApp function failing to successfully login](http://stackoverflow.com/questions/28794290/why-is-my-urlfetchapp-function-failing-to-successfully-login) – AdrianHHH Jun 04 '15 at 10:50

0 Answers0