Web response in C# .NET doesn't work more than a couple of times

Question

I am developing an application using twitter api and that involves writing a method to check if a user exists. Here is my code:

  public static bool checkUserExists(string user)
    {
        //string URL = "https://twitter.com/" + user.Trim();
        //string URL = "http://api.twitter.com/1/users/show.xml?screen_name=" + user.Trim();
        //string URL = "http://google.com/#hl=en&sclient=psy-ab&q=" + user.Trim();
        string URL = "http://api.twitter.com/1/users/show.json?screen_name=" + user.Trim();
        HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(URL);
        try
        {
            var webResponse = (HttpWebResponse)webRequest.GetResponse();
            return true;
        }
        //this part onwards does not matter
        catch (WebException ex)
        {
            if (ex.Status == WebExceptionStatus.ProtocolError && ex.Response != null)
            {
                var resp = (HttpWebResponse)ex.Response;
                if (resp.StatusCode == HttpStatusCode.NotFound)
                {
                    return false;
                }
                else
                {
                    throw new Exception("Unknown level 1 Exception", ex);
                }
            }
            else
            {
                throw new Exception("Unknown level 2 Exception", ex);
            }
        }
    }

The problem is, calling the method does not work(it doesn't get a response) more than 2 or 3 times, using any of the urls that have been commented, including the google search query(I thought it might be due to twitter API limit). On debug, it shows that it's stuck at:

var webResponse = (HttpWebResponse)webRequest.GetResponse();

Here's how I am calling it:

Console.WriteLine(TwitterFollowers.checkUserExists("handle1"));
Console.WriteLine(TwitterFollowers.checkUserExists("handle2"));
Console.WriteLine(TwitterFollowers.checkUserExists("handle3"));
Console.WriteLine(TwitterFollowers.checkUserExists("handle4"));
Console.WriteLine(TwitterFollowers.checkUserExists("handle5"));
Console.WriteLine(TwitterFollowers.checkUserExists("handle6"));

At most I get 2-3 lines of output. Could someone please point out what's wrong?

Update 1:
I sent 1 request every 15 seconds (well within limit) and it still causes an error. on the other hand, sending a request, closing the app and running it again works very well (on average accounts to 1 request every 5 seconds). The rate limit is 150 calls per hour Twitter FAQ.

Also, I did wait for a while, and got this exception at level 2: http://pastie.org/3897499

Update 2: Might sound surprising but if I run fiddler, it works perfectly. Regardless of whether I target this process or not!

You need to Dispose the response. If you don't, it leaves the connection open. Which most servers will block you if you have too many open connections to them. — Will, May 11 '12 at 23:09

McGarnagle · Answer 1 · 2012-05-11T22:46:43.010

The effect you're seeing is almost certainly due to rate-limit type policies on the Twitter API (multiple requests in quick succession). They keep a tight watch on how you're using their API: the first step is to check their terms of use and policies on rate limiting, and make sure you're in compliance.

Two things jump out at me:

You're hitting the API with multiple requests in rapid succession. Most REST APIs, including Google search, are not going to allow you to do that. These APIs are very visible targets, and it makes sense that they'd be pro-active about preventing denial-of-service attacks.
You don't have a User Agent specified in your request. Most APIs require you to send them a meaningful UA, as a way of helping them identify you.

I sent 1 request every 15 seconds (well within limit) and it still causes an error. on the other hand, sending a request, closing the app and running it again works very well (on average accounts to 1 request every 5 seconds). The rate limit is 150 calls per hour. — orderof1, May 11 '12 at 22:58

JamieSee · Accepted Answer · 2012-05-14T16:26:52.513

Note that you're dealing with unmanaged resources underneath your HttpWebResponse. So calling Dispose() in a timely fashion or wrapping the object in a using statement is not only wise, but important to avoid blocking.
Also, var is great for dealing with anonymous types, Linq query results, and such but it should not become a crutch. Why use var when you're well aware of the type? (i.e. you're already performing a cast to HttpWebResponse.)
Finally, services like this often limit the rate of connections per second and/or the number of simultaneous connections allowed to prevent abuse. By not disposing of your HttpWebResponse objects, you may be violating the permitted number of simultaneous connections. By querying too often you'd break the rate limit.

Web response in C# .NET doesn't work more than a couple of times

2 Answers2

Linked