The Lightstreamer Blog: Real-time Business Analytics Using Dojo and Lightstreamer

(Below is a re-post from the SitePen Blog)

The proliferation of online data gives businesses more information than they can possibly manage. Real-time dashboards and trading systems help users make sense of financial information, analytics, business trends, customer stats, and much more. The Dojo Toolkit and Lightstreamer are two popular choices available for building your own platforms and applications.

Building on a long-standing collaboration between SitePen and Lightstreamer (Weswit Srl), SitePen created the new LightstreamerStore. Based on the new Dojo Toolkit 1.6 Object Store API and the latest Lightstreamer preview releases, it is intended to make streaming data from a Lightstreamer server super-simple for developers. SitePen also updated the existing Lightstreamer + Dojo Toolkit demo to demonstrate this simplified interface. The demo visualizes the data returned in real-time from Lightstreamer using both the Dojo DataGrid and Dojo Charting. Charts display the “normalized last price” to more easily visualize changes in prices.

The new LightstreamerStore is planned for release in version 1.7 of the Dojo Toolkit. Preview versions are now available with the current development trunk of the Dojo Toolkit.

“We’re very excited to have worked with SitePen on this project,” said Alessandro Alinone, co-CEO and CTO at Lightstreamer. “StockList showcases the most powerful capabilities of both Dojo and Lightstreamer, showing what’s easily possible with a powerful real-time Comet server and an extremely flexible JavaScript toolkit”

Key Technologies

Using the Lightstreamer 4.0 preview Moderato release, SitePen used the new Dojo Object Store approach in conjunction with two of the most powerful technologies in the Dojo Toolkit–the DataGrid and the Charting library– to create a completely data-driven visualization dashboard application. No other technologies were hurt during the production of this project.

Obstacles

A number of obstacles were removed with this project:

Making sure communication and dispatching data on the client-side worked flawlessly
Ensuring the Dojo DataGrid updates based on both the items being displayed, and the speed at which data changes
Adapting the returned data from a “streaming” concept to an historical one
Accurately displaying changes in value on the Charts based on time received
Switching the value ranges on the Chart’s x-axis based on the latest data being received (i.e. “scrolling”)

The new LightstreamerStore combines the abilities of the Lightstreamer Web Client with Dojo’s own Promises and QueryResults to allow a user to quickly setup a LightstreamerStore and perform a query on it. The query itself then opens up a data stream, from which the results of the query initiates a function called “observable,” the basic entry point for consuming data with the Dojo Toolkit’s Object Stores. Each execution of “observable” tells the listening functions what object is being altered and the new data for the object – all in an easy-to-use JSON format.

From there, an adapter (dojo.data.ObjectStore) was used as the interface to feed data into the Dojo DataGrid. This allowed SitePen to use the standard data-binding interfaces between dojo.data and the Grid to allow for updating. In addition, a slight “throttle” was used to limit how often the DataGrid responded to updates; this was enabled to allow the Grid to respond to batches of updates as opposed to trying to handle each updated as it was received, improving application performance and behavior.

The charts were more challenging, in part because of the nature of the stream being returned by Lightstreamer’s server. Dojo Charting is typically used not just to visualize data, but to visualize it historically. In other words, it is more useful to visualize not just the latest data, but how it trends over time. The nature of the Stock List feed from Lightstreamer prevented SitePen from simply reading that stream and plotting the latest points, because a real-time view discards any historical data. SitePen created a set of items that listened to data being returned from the Lightstreamer server, and pushed them into a historical list–recording both the value received, and the timestamp when that data was received. From there, stock items that the user wished to visualize in a chart were passed to the chart engine as-is.

One problem this introduces is ensuring that the chart’s data was “scrolled” accurately, much like any other monitoring tools, such as Windows’ TaskManager shows CPU usage. Dojo’s Charting engine supports this through the use of “named axes.” If you keep track of the current time stream, you can update an axis (and all the data that is rendered by it) by simply re-adding an axis with the same name as the original, with new parameters. In the case of Lightstreamer, the new parameters were simply an update to the minimum and maximum values of the X axis. Thus, based on a 30 second window, the chart in the demonstration will continue to scroll as long as the server and client are communicating.