TL;DR Map full of tweets in real-time. Demo here.

Preamble

Ever since I learned of the Twitter Streaming API, I have been fascinated by it. Twitter's documentation for it is good although it doesn't provide many details on implementing it in a specific programming language such as `C#`. So I resolved to roll my own class library to connect to the `https://stream.twitter.com/1.1/statuses/filter.json` endpoint. Here is how I fared..

Defining expectations

I have previously created projects using the Tweetinvi C# wrapper for the Twitter Streaming API (tweetinvi.codeplex.com), and coverage of all endpoints is pretty good. The project's creator, known simply as Linvi to you and I, is very helpful and is a wunderkind with all things C#. The Tweetinvi library is quite complex, and in my opinion a little overcomplicated for what I was trying to achieve (simply open a long-standing HTTP connection to a streaming endpoint). A lot of the basic logic associated with lodging the long-running HTTP connection is abstracted out into a million interfaces and confused the hell out of me. And if you're using someone else's code, especially when it's in the form of a hulking great library, you don't really learn much about the internals do you? SO, I resolved to roll my own, with these caveats:

  • The current implementation should be as simple as possible. Preferably leaning towards the procedural for the meantime.
  • The logic for connecting to the API should be part of just one class, no interfaces.
  • I should be able to write the code necessary to make a successful connection to the API in less than 2 hours.

One feature of the Tweetinvi library that I really liked was the use of Event Handlers to indicate that a new tweet had been received from the API. This was kinda a "must have" when I came to implementing my version. It just makes sense given the real-time paradigm. I particularly liked the syntax to receive tweets in the client class (shown below):

filteredStream.MatchingTweetReceived += (sender, args) => 
{ 
   // do stuff to the individual tweet (i.e. args.Tweet)
};

Authorization / Authentication with the Twitter API

With the advent of version 1.1 of the API, basic authentication was deprecated and OAuth 1.0a was enforced. I found a useful base class which had a lot of the OAuth connection logic I would need to use when authenticating with Twitter hosted at the OAuth repository on Google Code (link). So the first step was to authorise via OAuth with the API:

_webRequest = (HttpWebRequest) WebRequest.Create(FilteredUrl);
_webRequest.Timeout = -1;
_webRequest.Headers.Add("Authorization", GetAuthHeader(FilteredUrl + "?" + postparameters));

The GetAuthHeader method builds upon the code from the OAuthBase class from the OAuth library, but provides a specific implementation for the Twitter API:

private string GetAuthHeader(string url)
{
    string normalizedString;
    string normalizeUrl;
    string timeStamp = GenerateTimeStamp();
    string nonce = GenerateNonce();
    string oauthSignature = GenerateSignature(new Uri(url), _consumerKey, _consumerSecret, _accessToken, _accessSecret, "POST", timeStamp, nonce, out normalizeUrl, out normalizedString);

    // build the request header
    const string headerFormat = "OAuth oauth_nonce="{0}", oauth_signature_method="{1}", " +
                                "oauth_timestamp="{2}", oauth_consumer_key="{3}", " +
                                "oauth_token="{4}", oauth_signature="{5}", " +
                                "oauth_version="{6}"";

    return string.Format(headerFormat,
        Uri.EscapeDataString(nonce),
        Uri.EscapeDataString(Hmacsha1SignatureType),
        Uri.EscapeDataString(timeStamp),
        Uri.EscapeDataString(_consumerKey),
        Uri.EscapeDataString(_accessToken),
        Uri.EscapeDataString(oauthSignature),
        Uri.EscapeDataString(OAuthVersion));
}

Twitter's documentation specifies the format that the Authorisation header takes. Here is an example from their documentation (link):

OAuth oauth_consumer_key="xvz1evFS4wEEPTGEFPHBog", oauth_nonce="kYjzVBB8Y0ZFabxSWbWovY3uYSQ2pTgmZeNu2VS4cg", oauth_signature="tnnArxj06cWHq44gCs1OSKk%2FjLY%3D", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1318622958", oauth_token="370773112-GmHxMAgYyLbNEtIKZeRNFsMKPR9EyMZeS9weJAEb", oauth_version="1.0"

One key thing here to take note of is that the values are url-encoded. As you can see from the method above, I am using `Uri.EscapeDataString` to encoded values.

Fulfilling the HTTP Request and Response pattern in real time

So once I'd successfully authorized with the Twitter API via OAuth, I needed to figure out a way of creating the long standing HTTP connection. The Twitter API docs note:

Establishing a connection to the streaming APIs means making a very long lived HTTP request, and parsing the response incrementally. Conceptually, you can think of it as downloading an infinitely long file over HTTP.

So how do we "parse the response incrementally" using `C#`? I did some research into the `HttpWebRequest` class and found the `BeginGetResponse` method to be of interest. The MSDN documentation describes the method as "[beginning] an asynchronous request to an Internet resource." So I did some experimentation and came up with the below bit of code:

_webRequest.BeginGetResponse(ar =>
{
    // Get async request state of incremental block
    var req = (WebRequest) ar.AsyncState;
    
    // Get the proper response obj for the current response block "increment"
    using (var response = req.EndGetResponse(ar))
    {
        // Use a StreamReader to read the response stream for the current incremental block
        using (var reader = new StreamReader(response.GetResponseStream()))
        {
            while (!reader.EndOfStream)
            {
                // Deserialize the JSON obj to a POCO automatically with Newtonsoft JSON.NET
                var jsonObj = JsonConvert.DeserializeObject<Tweet>(reader.ReadLine(),
                new JsonSerializerSettings());

                // Raise tweet received event - this is remarked upon later in the article
                Raise(TweetReceivedEvent, new TweetEventArgs(jsonObj));
            }
        }
    }

}, _webRequest);

Now I'm sure that there is a better way to achieve this but for now this'll do! Anyway so once all of the boilerplate code to lodge web requests and receive responses from the Twitter API, it was as simple as using `JsonConvert.DeserializeObject<T>();` to turn JSON into a POCO (Plain Ole' C# Object).

Modeling the tweet data

I decided to use Newtonsoft JSON.NET to deserialize the JSON string representing each Tweet. I created a custom Tweet model class with a flat implementation (i.e. no nested sub classes or anything else) and I used the `JsonProperty` attribute to mark each instance field with the correct JSON property as found in the original JSON source returned from the API:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.Text;
using System.Threading.Tasks;
using Newtonsoft.Json;

namespace TwitterClient.Infrastructure.Models
{
    public class Tweet
    {
        [JsonProperty("text")]
        public string Text;

        [JsonProperty("id_str")]
        public long Id;

        [JsonProperty("screen_name")]
        public string UserName;

        public override string ToString()
        {
            var sb = new StringBuilder();
            sb.Append(Text);
            return sb.ToString();
        }
    }
}

UPDATE: I removed the old model above and replaced it with an auto-generated set of models based on the JSON string from a sample tweet. I used the wonderful json2csharp (http://json2csharp.com/) tool to do this. The final set of models can be found here.

Raising a "Tweet Received" event

The final piece in the puzzle was to raise an event using an event handler that a client could subscribe to. The boilerplate code for this involves registering a new `event` and a instantiating a `delegate` instance which provides the promise of an implementation:

public class TwitterStreamClient : OAuthBase
{
    public event TweetReceivedHandler TweetReceivedEvent;

    public delegate void TweetReceivedHandler(TwitterStreamClient s, TweetEventArgs e);

    public TwitterStreamClient() {

    	// Make a fake tweet to test the event being raised
    	Tweet t = new Tweet() {
    	     Text = "A dummy tweet"
    	};

    	// Raise the new event so that clients can receive notification of the event
    	Raise(this, new TweetEventArgs(t));

    }
    public void Raise(TweetReceivedHandler handler, TweetEventArgs e)
    {
    	// check that the event handler has any subscribers
        if (handler != null)
        {
            handler(this, e);
        }
    }
}

In order to receive notification of the event in the caller class, you must subscribe to the event before calling the `Start()` method which raises the event (otherwise the handler will be `null`). I created a simple Console application to test my newly created class library:

class Program
{
    static void Main(string[] args)
    {
        string accessToken = "XXXX"; // your access token
        string accessTokenSecret = "XXXX"; // your access token secret
        string customerKey = "XXXX"; // your consumer key
        string customerSecret = "XXXX"; // your consumer secret

        var config = new StreamConfig()
        {
            ConsumerKey = customerKey,
            ConsumerSecret = customerSecret,
            AccessToken = accessToken,
            AccessSecret = accessTokenSecret,
            GeoOnly = true
        };
        var stream = new TwitterStreamClient(config);

        // subscribe to the event handler first
        stream.TweetReceivedEvent += (sender, tweet) =>
        {
            if (tweet.Tweet.Id != 0)
            {
                Console.WriteLine(tweet.Tweet.Id);
            }
        };
        stream.Start(); // Start the stream process
    }
}

Tying it all together in a sample MVC app

In order to truly show off the awesomeness of the Twitter API, let's create a test MVC app with a map and broadcast tweets onto it in real-time. Now one of the main problems we are going to encounter when accessing the stream in a web app context is that whereas in a Console application, there is a single point of entry to executing the program (e.g. you press the "run" button and the application begins execution), in a web application, multiple clients may be connecting at the same time, each time triggering common code.

One of the constraints of the Twitter API is that only one or two streams can be open to Twitter at any time per set of application credentials, so therefore creating a new streaming connection instance every time a new user logs onto our site is a massive no-no.

Instead, I needed to figure out a way of putting the logic associated with connecting to Twitter in a separate logical context to the code concerned with processing HTTP requests and responses for each user. In an ASP.NET application, the obvious event is `Application_Start` in the `Global.asax` file as this is called at the beginning of the lifetime of the application. So I resolved to start a new instance of a wrapper class I created `TwitterConnection` using `Task.Factory.StartNew()`. Stephen Cleary warns of `Task.Factory.StartNew` usage in his article here, so I will probably convert this code to use `Task.Run` in the immediate future. Anyway, here is my full Global.asax file:

using System.Threading.Tasks;
using System.Web;
using System.Web.Mvc;
using System.Web.Optimization;
using System.Web.Routing;

namespace TwitterClient.Web
{
    public class MvcApplication : HttpApplication
    {
        protected void Application_Start()
        {
            AreaRegistration.RegisterAllAreas();
            FilterConfig.RegisterGlobalFilters(GlobalFilters.Filters);
            RouteConfig.RegisterRoutes(RouteTable.Routes);
            BundleConfig.RegisterBundles(BundleTable.Bundles);
            
            // Should probably use Task.Run()
            Task.Factory.StartNew(() => new TwitterConnection());
        }
    }
}

Where the `TwitterConnection` class starts up an instance of the stream logic found in my class library:

public class TwitterConnection
    {
        // Consumer Keys + Secrets
        private readonly string _consumerKey = ConfigurationManager.AppSettings.Get("consumerKey");
        private readonly string _consumerSecret = ConfigurationManager.AppSettings.Get("consumerSecret");
        // Twitter OAuth Credentials
        private readonly string _accessKey = ConfigurationManager.AppSettings.Get("accessToken");
        private readonly string _accessToken = ConfigurationManager.AppSettings.Get("accessTokenSecret");

        private readonly IHubContext _context;

        public TwitterConnection()
        {
            _context = GlobalHost.ConnectionManager.GetHubContext<TwitterHub>();
            var config = new StreamConfig()
            {
                ConsumerKey = _consumerKey,
                ConsumerSecret = _consumerSecret,
                AccessToken = _accessKey,
                AccessSecret = _accessToken,
                GeoOnly = true
            };
            var stream = new TwitterStreamClient(config);
            
            // Subscribe to the tweet received event
            stream.TweetReceivedEvent += (sender, args) =>
            {
                // Broadcast the tweet to the client-side
                _context.Clients.All.broadcast(args.Tweet);
            };
            stream.Start();     
        }
    }

On the client-side

We will need to use the Google Maps JavaScript API to do this, so we need to include the reference to it in our `_Layout.cshtml` layout file:

<script type="text/javascript" src="https://maps.googleapis.com/maps/api/js"></script>

And we also need to add a `map` element to the individual `View`:

<div id="map-canvas"></div>

The final step is to written a little bit of JavaScript to set up the map and the SignalR client-side connection to the Hub proxy:

var map;
function initialize() {
    var mapOptions = {
        center: { lat: -34.397, lng: 150.644 },
        zoom: 8
    };
    map = new google.maps.Map(document.getElementById('map-canvas'),
        mapOptions);
}
// add window.load listener to call initialize function
google.maps.event.addDomListener(window, 'load', initialize);

$(function () {
    // obtain reference to the hub proxy and hub itself
    var theHub = $.connection.twitterHub;

    // this is the function that the server will call to broadcast new tweets
    theHub.client.broadcast = function (tweet) {

        var c = tweet.coordinates.coordinates;

        if (!$.isEmptyObject(tweet.coordinates)) {
            // To add the marker to the map, use the 'map' property
            var marker = new google.maps.Marker({
                position: { lat: c[1], lng: c[0] },
                map: map,
                title: tweet.text
            });
            var infowindow = new google.maps.InfoWindow({
                content: tweet.text
            });
            google.maps.event.addListener(marker, 'click', function () {
                infowindow.open(map, marker);
            });
        }       
    };
});

Next steps:

  • Implement await in async methods and convert synchronous methods across to use `async` and `await`
  • Clean up the code and abstract out icky procedural code

Demo

Full working demo is available at http://twitterstreammap.azurewebsites.net/.

Download

The full source code for this project can be found at https://github.com/adaam2/APoorMansTwitterStreamingClient/, along with an MVC example using SignalR to stream tweets in real-time to the page.