Push Technology, Comet, and WebSockets: 10 years of history from Lightstreamer's perspective

By Alessandro Alinone (originally published on Comet Daily)

More than ten years have passed since the creation of Lightstreamer. Now that Lightstreamer 4.0 is generally available, it is a good moment to look back at what happened in the history of Push/Comet and to share a short analysis of the current trends from Lightstreamer’s perspective.


1996-2000: The first wave of Push Technology (Webcasting)

At that time, Push Technology was mainly referred to techniques also known as webcasting, narrowcasting, or channeling. A channel was related to some category of information, and once the user registered with one or more channels, she would automatically receive the information, which was displayed by dedicated client software (thick applications, browser plug-ins, or special screen savers). In 1996, PointCast, the first push system based on channels, was created. Soon after, over thirty players entered this market, including Microsoft and Netscape. Push Technology was expected to become a killer application, but this forecast did not come true, and in 2000 that kind of Push Technology was finally dead. The main reason is that the pushed information was very coarse-grained and not real-time. Users did not need to find significant amounts of information downloaded to their computer every morning, most of which they would never read. Someone compared the first wave of Push Technology to having giant heaps of newspapers dumped on your doorstep every morning…

2000 onwards: The second wave of Push Technology (Comet)

Starting in 2000, the success of online security trading systems created a new need for real-time data being pushed to the user’s browser. The information had to be very fine-grained (at the level of a single field being updated) and very real-time (the lower the latency between the data generation and its delivery, the better the quality of the trading platform). The first players in this arena were Caplin and Lightstreamer, together with Pushlets and KnowNow. In these eleven years, Push Technology has kept evolving in many ways: by adding new features to the available solutions; by improving reliability and performance; by seeing many new entrants coming into this market niche; by proposing new standards.

The success of this wave is increasing, with more and more production systems taking benefit from the real-time data delivery. Most of the financial trading platforms currently employ some form of data push. Online auctions, betting, and gaming systems are moving in the same direction. And we are seeing several new projects in both the military and aerospace domains leveraging this wave of Push Technology.

The very dynamic nature of this set of technologies has made it very difficult to agree even on a common umbrella term. Below is a list of the many terms that can be somehow related to Push Technology (I am fully aware of the many, more or less subtle differences in place among such words):

• Push Technology
• Web Streaming
• Web Push
• Streaming AJAX
• Reverse AJAX
• AJAX Push
• Comet
• HTTP Streaming
• HTTP Long Polling
• Real-Time Web
• Last Mile Messaging
• Internet Messaging
• Bi-directional HTTP
• Full-Duplex Web Communication
• WebSocket

How about Comet? When Alex Russell coined the Comet term in 2006, it seemed to be a very good umbrella word and we all started to adopt it. But then, partly due to marketing-originated differentiation needs, and partly due to actual technological differences, other terms kept emerging or being reused. In particular, starting from 2007, the interest began to focus on bi-directional communication, leading to a new wave of Push Technology (see next section). Notice that the list above tries to cover both these waves, as the boundaries between the two are very blurry and arbitrary.
The push channel (from server to client) can be implemented via several different techniques, such as:

• Polling
• Long Polling
• Frame Streaming
• Iframe Streaming
• Flash Streaming
• XHR Streaming
• Server-Sent Events

Several protocols have been created in the past few years employing one or more of the techniques above. Some of them derive from open-source initiatives (e.g. Bayeux and BOSH) or from standard specifications (HTML5 Server-Sent Events), others (like Lightstreamer’s protocol) are proprietary. Even if Lightstreamer’s network protocol is publicly documented, we prefer to provide high-level APIs for each supported client-side technologies, including mobile applications. This way, we can keep full control over the heuristics mechanisms needed to choose the best delivery technique to use, including low-latency fall-back processes.

2007 onwards: The third wave of Push Technology (WebSocket)

Bi-directional communication means not only pushing real-time messages from the server to the client, but also the reverse. This may seem awkward, as sending messages from the client to the server at the client’s discretion has always been the normal behavior of HTTP! In reality, the aspect that is considered new and important has been the ability to send such reverse messages with low latency and high frequency. In most cases HTTP can stream messages with low latency and high frequency from the server to the client, but it always requires a full round-trip to send messages from the client to the server. This is a constraint imposed by how browsers and proxies are implemented. And this is one of the reasons that led to the WebSocket specification, aimed at enabling full-duplex communication over a single TCP connection (the other main reason being to prevent intermediaries from buffering the streaming content, as it happens in some cases with HTTP).

At the time of writing [July 2011; see update at the end], the WebSocket specification is still in draft status (the latest version is draft-ietf-hybi-thewebsocketprotocol-09) and its adoption by “infrastructure” vendors (including browsers, proxies, packet inspectors, antivirus’, etc.) is still extremely immature and fragmented.

Without WebSocket, the reverse push channel (from client to server) can be implemented on the top of a second HTTP connection. The major problem here is that the application code has no control over the binding between HTTP requests and HTTP connections, which are handled by a pool manager that is part of the browser. This means that the browser might decide to send two backward messages originated from the application code on the top of two different physical TCP connections, with the risk of altering the message order. This issue, together with the low-latency and high-frequency requirements mentioned above, originated the need for a sophisticated reverse channel implementation within the application layer, to guarantee message ordering, low latency, and high frequency.
For example, Lightstreamer implements all of the mechanisms needed to give the developer a high-level and robust abstraction over the HTTP reverse channel. In particular, Lightstreamer automatically manages message numbering and re-ordering, transparently batches messages to minimize the round trips, and implements guaranteed delivery, by means of acknowledgements and automatic retransmissions.

WebSocket simplifies the implementation of both the communication channels, but WebSocket is only a transport layer, over which application-level protocols need to be implemented. For example, most Push Technology solutions are based on publish-and-subscribe paradigms. WebSocket alone does not offer any pub/sub facility, which must be implemented on the top of it. Another example: WebSocket alone does not dictate any technique for throttling the data flow of filtering/conflating the data based on bandwidth constraints or on Internet congestion. Again, this is something left to a higher-level implementation. For these reasons, eleven years of experience originated from second-wave Push Technology will be invaluably precious for the success of the third wave. Furthermore, the transition between the two waves requires some years of co-existence, until WebSocket is: a) specified in final form; b) fully deployed across all infrastructure components, implying the abandonment of all the older browsers (for instance, at the time of writing, Microsoft’s Internet Explorer requires a separate download to support the current draft of WebSockets on both IE9 and IE10 beta).

As for Lightstreamer, we have been delaying the roll-out of WebSocket, waiting for the more stable specifications and better browser support. Now, we believe that the time has finally come to inject all the experience gained in the past years into the third wave of Push Technology. Our role will be to guarantee a smooth and safe transition between the waves, taking care of all the issues above.

Conclusion

Our perspective on the history of the technologies aimed at enabling the Real-Time Web (known under many different names and with different meanings, such as Push Technology, Comet, or WebSocket) will help with the new third wave that is centered around bi-directional communication in general, and the WebSocket standard in particular. I maintain that WebSocket needs to leverage the second wave of technology to be successful, due to: the necessity, for several more years, to be able to fall back to high quality Comet when needed, and the necessity to build application-level protocols and network management optimizations over WebSocket that have already been implemented and deployed on solid second-wave solutions.

UPDATE: The WebSocket Protocol staged up for final IETF publication as an Internet Standard (RFC 6455) at the end of November 2011. Full WebSocket support has been implemented as part of Lightstreamer 5.0, codename Colosseo, made publicly available via a preview release in April 2012.

UPDATE: A slide deck is available convering some of the topics above:
http://www.slideshare.net/alinone/from-push-technology-to-the-realtime-web