In my last post I talked about how we captured and stored the tweets pertaining to the Olympic Games. In this post I’m going to explain how we provide an API to the data we will subsequently calculate.
Then we ensure that a call to the root of the server is transferred to the root of version 1 of our API. This versioning of the API allows us to make changes in the future, should we want to, without breaking the current client.
Next we hook up any call to the root so that it answers a JSON object describing all the graphs that the API will create, along with the end point for each graph. This ensures that any client knows exactly what our API will produce and how to consume all, or a sub set, of the available graphs.
NodeJS is non-blocking in nature, but it is also single threaded and if a CPU intensive calculation is being carried out, then the thread will block and no further requests from clients will be serviced. We get around this limitation by abstracting the calculation away from NodeJS and spinning off another process to complete it. In this case, the calculation is re-written in Python. For those that require it, a good refresher on the difference between threads and processes can be found here.
Traditionally there has been a problem with communicating between processes, as they do not share memory. NodeJS fixes this problem in two ways, firstly via channels and secondly, by the method best suited to our needs, by allowing the callback, passed as an argument to the “exec” function, to have access to the stdout of the spun process. Thus, all we need do is to issue a “print” statement from our Python code, to have the data made available to us in our parent process. We also have access to “error” which means we will be informed if an error is raised in our spun process. All this allows us to use a pattern whereby we check for the presence of an error, if we find one, we return a 501 HTTP status code, if we do not, we return a JSON representation of the spun process’s “stdout”, along with a 200 HTTP status code.
That’s it for this post, in the next we’ll take a look at how we calculated some of the measures you see at: http://bit.ly/olympicStats