MongoDB is part of the new breed of N0SQL databases which offer more flexibility over a traditional relational database management system (RDBMS). It uses a JSON based query format, a dynamic schema and a simplified means to retrieve data from multiple documents.
In this tutorial, we’ll be demonstrating an integration of MongoDB and PubNub. We’ll build a simple stock ticker application where price updates of the key stock market indices around the world will be stored in a MongoDB database.
And using the PubNub Data Stream SDK, we will build a server and a client application to broadcast and display these stock prices.
Further, the client application can also fetch historical prices and display a chart. In line with PubNub’s real-time capabilities, the client application will always automatically update the display with the latest price updates without needing a manual refresh.
Real-time MongoDB Tutorial Considerations
Before we dive into the nuances of application logic, here are a few considerations for our project:
- The stock market price data is randomly generated
- The database is initially populated with some arbitrary price data
For the purpose of this demo, the server application is built on Python which interfaces with the DB and also publishes stock data on a global broadcast channel. The client application is a front-end JavaScript UI application which subscribes to global broadcast channel and updates its display with the latest price whenever there is a price change.
See the illustration below to understand the high level architecture of this application.
Additionally, the server also subscribes to a global listen-only channel for receiving requests from clients for the stock price history data. A client can request for price history by publishing a request on the global listen-only channel and it can display a chart showing price history trend of an index. Each client opens a separate private channel for receiving price history data from server.
Project Libraries
We will use the following libraries for this application:
- Python bindings for interfacing with MongoDB Server: Pymongo
- Python SDK for PubNub at the server: PubNub Python SDK
- Javascript SDK for PubNub at the client: PubNub Javascript SDK
Set Up
The key working aspects of the application are as follows
- Server
- MongoDB server setup and data format
- Simulation of stock indices at random intervals with arbitrary values
- Broadcasting and client request handling via PubNub channels
- Client
2.1 Client application initialization based on PubNub Javascript API
2.2 Price dashboard UI
2.3 Price history chart display
1. Server
1.1 Real-time MongoDB server setup and data format
First things first, we need a working MongoDB setup where the data can be stored and retrieved. Fortunately, mongoDB offers a cloud service with free access for a limited number of servers. You can create an account, install the mongoDB agent on your computer and you have the database server ready. The mongoDB agent acts as a proxy to forward your requests to the cloud hosted MongoDB server under your MMS account.
Now with the database server up and running, we need to define the format (schema) in which the stock data will be stored. For this application lets consider a few of the popular stock market indices from around the world.
- Dow Jones
- NASDAQ
- FTSE
- NIKKEI
The server will be constantly monitoring the prices of these indices and broadcast them on the global broadcast channel.
For the benefit of those who come from RDBMS world, here is a quick low down on the terminologies used by MongoDB. Lets clarify this before we decide on the data schema format.
- In MongoDB , all related data is aggregated as a “collection”. This is the equivalent of “Table” in RDBMS.
- In MongoDB, the data is stored within a collection as documents. Each document is equivalent of a record (or row) in the Table of a RDBMS.
- Unlike in RDBMS systems where all tables have to have a predefined schema , MongoDB does not required this. Moreover each document in a MongoDB collection can have its own schema format. So in terms of RDBMS parlance this means that every row on a MongoDB Table can have its own schema and this can vary from row to row. Quite Dynamic !!
- Database updates ( inserts, update, delete) on MongoDB are done using the JSON format which is comparable with SQL and very easy to read and understand.
For more details on the MongoDB database operations, refer the MongoDB Manual.
So assuming we define a database named “stockdb” and a collection named “stockcollection” , the schema definition format and type for each field within the collection stockcollection will be something like this.
S No. Field Name Field Value
1 name string
2 value string
3 change string
4 time numeric
Above: MongoDB Document Schema Format for Individual Index Prices
Here is a brief description of each of the fields
- name: Name of the stock index
- value: Current price value
- change: Delta change from the previous value
- time: Time of change (as per UNIX time format)
Here is how we can provision this sample data in database via the PyMongo API.
Connect to db and create the database and collection
mongoconn = pymongo.MongoClient(server,port) #Create MongoDB Client
db = mongoconn.stockdb #Create db named stockdb
coll = db.stockcollection #Create collection stockcollection
Insert a few documents with sample stock prices
stock = {'name':'NASDAQ' , 'value':4630.60 , 'change':'+6.06' , 'time' : 1}
coll.insert(stock) #Insert document
Typical queries on document
coll.find({ ‘name’ : ‘NASDAQ’}) #Query for all records of ‘NASDAQ’
1.2 Simulation of stock indices at random intervals with arbitrary values
Since we do not have access to the real stock market feed, we are going to simulate this by having a python code which generates and populate stock prices at random intervals within 10 seconds.
Check the GitHub source below for python code to generate random index prices:
1.3 Broadcasting and client request handling via PubNub channels
The server application has two major tasks:
- Broadcasting the latest price update for each index
- Serving client requests for historical price data
Broadcasting the latest price update for each index
Broadcasting the latest price update is done as part of the server’s main loop. After each price update the server broadcasts the data in the following JSON format defined as per the figure above.
{
“name” :”NASDAQ” ,
“value” : “4638.20” ,
“change” : “+11.25” ,
“time” : 1412322567
}
Above: Price Broadcast Format
Here is the python code which does all the magic. We have to use the Python API for MongoDB connection and PubNub Python API for
Serving client requests for historical price data
If a client requests for historical price data, the server needs to honour that. For this, it subscribes to the global listen-only channel. The name for this channel is ‘stockhistory’.
The client will send the request in the following JSON format:
{
“name” : <name of the <index>,
“backtime” : < Time in minutes>,
“channel” : <channel name>
}
Above: Price History Request Format
- name: Specifies the name of the index whose price history is requested
- backtime: Numeric value n , indicating the last n minutes from where the historical data will be fetched.
- channel: The private channel name of the client (where server should publish the data)
From a security standpoint, it may not be a good practice to send the channel name directly within data but for this demo application, it suffices the functionality.
The server, upon receiving this request first fetches all the document records from MongoDB database based on the backtime. The data is formatted as JSON array and published on the client’s private channel as per the following format
[
{
“name” :”NASDAQ” ,
“value” : “4646.20” ,
“change” : “-2.00” ,
“time” : 1412322567
} ,
{
“name” :”NASDAQ” ,
“value” : “4648.20” ,
“change” : “+11.25” ,
“time” : 1412322534
},
{
“name” :”NASDAQ” ,
“value” : “4638.20” ,
“change” : “+10.00” ,
“time” : 1412322504
}
]
Above: Historical Price Response Format
This is handled in a separate python thread named ClientListenerThread. Here is the complete code within ClientListenerThread for handling and processing the client requests and fetching the data from MongoDB.
2. Real-time MongoDB Client
With the server part taken care of, we now have a stream of index prices which are regularly published over PubNub. So lets build a small web front end to consume this data and present it in a small dashboard.
Above: Client App
2.1 Client application initialization based on PubNub JavaScript API
We will have a JavaScript driven web application, built on top of PubNub’s JavaScript SDK. The app will display the current price of all the four indices and provide an option for displaying the historical price chart for any index.
But first we need the app to have a capability to publish to and subscribe for PubNub channels.
Here is how we can initialize the PubNub Javascript SDK:
We also need to define a callback function in javascript which gets triggered in the client when a new index price is published from the server.
Let’s call it updatePrice:
We will come back to updatePrice function and write the nuances of it, but for now let’s just hook it up to javascript SDK for subscribing to PubNub channels.
This call ensures that whenever the server publishes a message is on the channel ‘stockdata’ , the javascript SDK will invoke the updatePrice function passing the published message as msg argument.
According to our design , every client app will also have a private channel for receiving historical price data. So, as part of the app initialization we will also create a random channel name and store it as our private channel. PubNub’s API offers an alternative for this.
The uuid() call will generate a unique name for the channel which can be used for receiving price history data.
Just like we have updatePrice function which acts as a callback for individual index price updates, we also need another callback for price history updates. Lets call it updateChart.
And finally we hook it up with the JavaScript SDK for subscribing to PubNub channels.
With this we have initialized the PubNub SDK inside our app, have setup the subscription for channels and we also have callbacks defined for taking actions whenever we have a feed arriving on these channels. We are all set now, so lets build the front end HTML portion of our app.
2.2 Price dashboard UI
As depicted in the figure above, the price update for each index will be a row with elements which form a column like structure to display each index and its associated parameters, i.e. the price , price delta value and last update time.
Each price counter is housed inside an enclosing <div> styled to form the row and the columns.
The id attribute of the div identifies with the index name whose price it displays. This helps in easy identification of the html elements for updating the price.
Here is the main HTML code to display the price dashboard:
When the app is launched and ready, each price update arriving at the global ‘stockdata’ channel channel hits the callback updatePrice. So lets see how can we update the html DOM to display the latest price:
data = $.parseJSON(msg)
$(‘#’ + data[‘name’]).html(data[‘value’])
The data arriving at the PubNub channel is in JSON string format. First it is converted it to Javascript object and then we can access the individual fields of the object to get the actual data (See Figure above for the format of JSON string broadcasted from the server ).
Similarly the price delta and time is also updated.
See the complete code below:
Now whenever a new price is published on ‘stockdata’ channel, updatePrice function will spring into action and update the dashboard UI with the latest price.
With this our stock ticker application has come to life and we can now see the price counters getting updated live on the display.
2.3 Price history chart display
The app also has a capability to display price trends in the form of a miniature chart. For this purpose we will use the JQuery sparklines library.
The price trends are based on historical price data and the request for historical prices is made by clicking on the ‘Trend’ Button. When the server responds on client’s private channel. The callback function updateChart is triggered to update the DOM with the chart values corresponding to historical price values.
(See figure above for the format of JSON string containing historical prices)
See the complete javascript code below for chart update and display:
Here is how the price trends will look like on the client app:
Above: Client App with trend chart based on price history
For this application, the client always requests for last 30 mins of price data or last 60 price samples, whichever is less. However it can be easily extended to allow the user to select these parameters. This is left as an exercise for the reader.
Wrapping Up Real-time MongoDB
With the price history display taken care of, our client app’s functionality is complete. We can now either see the index price counter or the historical trend for each index separately.
What we have built in this tutorial is a bare minimal application and the idea is to demonstrate how can we trigger MongoDB operations either in conjunction with or as part of PubNub SDKs. Obviously, a lot more can be achieved and I hope that this demo will give a good starting ground for developers to create more meaningful MongoDB & PubNub based apps with more features and customization options.
Get the complete source of this application at GitHub. For more details on the PubNub SDKs used in this tutorial, visit the PubNub Python SDK and PubNub Javascript SDK documentation.
Enjoy!