Abstract
This article shows how to quickly and easily present continuous test results in a bar chart using ChartJS.
Motivation
I recently helped our IT department debug and fix a networking problem in a set of lab test machines. Over several weeks, I wrote test scripts which continuously checked the status of the networking problem.
As I steadily grew the complexity of the scripts, I found myself spending more and more time answering questions from the IT team, manually interpreting the results in my log files.
When it hit my threshold of pain, I searched and found an easy way to present the results visually. This way, the IT team could browse and view results themselves, on-demand, without bothering me. Score!
Technology Search
Fixing the networking problem was an ad-hoc effort, so I did not have a formal continuous-test framework with fancy dashboard where I could post the data. I needed to set up my own.
I researched a few data warehouse and data mining products and services, but they were too complicated for my simple needs.
Eventually, I happened upon Chart.js. I hadn't known about it or used it before. It's great.
What is Chart.js?
Chart.js is a small javascript library file. It is available as a free opensource project under the MIT license at http://www.chartjs.org/
To use it, you write some HTML and javascript, provide your data in JSON format, and ChartJS will render your data in a bar chart or other type of graph which you specify.
All you need is a little DIY scripting, an apache web server, and a browser.
References
I learned everything I needed to know from the official docs:
http://www.chartjs.org/docs/
And I found a perfect tutorial for my needs:
http://www.techrepublic.com/blog/software-engineer/chartjs-library-simplifies-adding-charts-to-web-applications/
Getting started
I recommend you download Chart.js and replicate the examples in the tutorial. Figure out how it works, and create a chart you like in an HTML file with canned data. Then you can move on to updating the data automatically...
Overview
Here is how it all works...
Now let's put the pieces together...
JSON test results
The first thing I did was enhance my test scripting to output its results in JSON format. It was already logging these numbers; the change was to also write them to a new JSON file.
For my purposes, I stored data in this format (only three samples shown for simplicity, and sanitized for privacy):
[
{ "time":"2014-0306-1801", "rc":{ "ok":11, "error0":0, "error1":0 } },
{ "time":"2014-0306-1935", "rc":{ "ok":8, "error0":2, "error1":1 } },
{ "time":"2014-0306-2029", "rc":{ "ok":11, "error0":0, "error1":0 } },
To bootstrap things, I manually created a flat file named results.json. I typed the opening square bracket for a JSON list, and saved the file.
Each time my test scripting finished running a test cycle, it appended a new line to the bottom of the file. My test scripts were written in the bash language, so my code looked like this.
JSON_STATS=" { \"time\":\"${DATE_NOW}\", \"rc\":{ \"ok\":${OK}, \"error0\":${ERROR0}, \"error1\":${ERROR1} } },"
echo "${JSON_STATS}" >> ${JSON_FILE}
You can do this with whatever scripting language you like, and I'm sure it's easier than bash.
Note that I *appended* the results to the file. This preserves history of earlier runs.
Also, JSON afficiandos will note that each line ends with a comma, and there is no closing square bracket to terminate the JSON list. In its present form, this means the file contents are not correctly-parseable JSON. Not to worry, this was intentional to allow easy appending. It is handled gracefully in the next step...
Conversion from JSON to HTML
Each time my test script appends new results to the JSON file, it calls a python program I wrote to convert the data from JSON to HTML.
There are two pieces of this: swizzling the results data for use by Chart.js, and creation of the HTML file...
Swizzle the JSON data
The output from my testcase scripting is organized according to time. That is, each line contains a timestamp along with results for that period of time.
However, for a bar chart, Chart.js requires the data to be organized differently. It requires a list of labels to be shown along the bottom of the chart, along with lists of data for each bar on the chart.
To do this conversion, the python program opened and read file results.json and swizzled the data.
To accommodate the fact that my results.json file does not contain correctly-parseable JSON, the python script first read in all the lines of the file in string form, deleted the trailing comma, and appended a closing square bracket. Voila, instant parseable JSON.
It handed the string to simplejson which parsed it and converted it to a list of objects.
jsonList = simplejson.loads( jsonString )
Then it swizzled the data organization as required by Chart.js. Using the data in my previous example, it looked like this after conversion:
timeList = [ "2014-0306-1801", "2014-0306-1935", "2014-0306-2029" ]
okList = [ 11, 8, 11 ]
error0List = [ 0, 2, 0 ]
error1List = [ 0, 1, 0 ]
Also, since my application was designed to present the most recent results of continuously-running tests, I chose to find and convert only the most recent data points, rather than everything in the history. (This example shows three; my real program showed 32.)
The same python program then created an HTML file named results.html.
Create HTML file
After a bit of experimenting with Chart.js using a manually-created HTML file, I settled on a design for a bar chart which I liked. I then converted the HTML file to be used as a template by replacing each list of data with a unique uppercase string.
That is, I removed the label list data and inserted the string TIME_LIST. I removed each data list in the datasets section, and replaced them with OK_LIST, ERROR_0_LIST, and ERROR_1_LIST.
The data portion of my HTML template file looked like this:
var data = {
"labels": TIME_LIST,
"datasets": [
{
"fillColor": "rgba(0,255,0,1)",
"strokeColor": "rgba(0,255,0,1)",
"data": OK_LIST
},
{
"fillColor": "rgba(255,0,0,1)",
"strokeColor": "rgba(255,0,0,1)",
"data": ERROR_0_LIST
},
{
"fillColor": "rgba(128,128,256,1)",
"strokeColor": "rgba(128,128,256,1)",
"data": ERROR_1_LIST
}
]
}
To convert the uppercase strings, the python program read the entire template file to a string (named templateString) and replaced each uppercase string with the real data. My python code looked like this:
templateString = templateString.replace("TIME_LIST", simplejson.dumps(timeList))
templateString = templateString.replace("OK_LIST", simplejson.dumps(okList))
templateString = templateString.replace("ERROR_0_LIST", simplejson.dumps(error0List))
templateString = templateString.replace("ERROR_1_LIST", simplejson.dumps(error1List))
The new HTML file was ready. The python program saved the new file as results.html.
Publishing on apache web server
After calling the python program to convert results.json to results.html, the last step for my bash test script was to transfer the HTML file to an apache web server. Each time it ran, it replaced the previous file on the web server.
Because this was on a private internal network, I transferred the file using linux utilities sshpass and scp.
sshpass -p${PASWORD} scp results.html ${USER}@${HOSTNAME}:/var/www/
Monitoring results
My partners in the IT team were now able to browse to the results.html file and see current results whenever they liked. They could tell how well any fixes they applied overnight were working througout the next day, and I was not required lose sleep to support them. Yay.
Here is a colorful example showing lots of errors (note: I changed the contents of the chart frequently during the debug cycle; this version of the chart only contained two bar columns, red and green).
Bells and whistles
After everything was working, I added a good old 'refresh' tag to the HTML head section. With this, the observer could leave the web page up in his browser, and it would refresh itself every two minutes without having to manually click refresh. Bonus!
<META http-equiv="refresh" content="120">
Closing
That's all there is.
It may seem complicated described in so many words, but its' really not. After several false starts with other technologies, developing all of this took me less than four hours, all told.
My IT team was delighted with it. I saved myself time by developing it. And I can use the system for other applications in the future. Success!
Take a look at Chart.js. You'll like it.
PS: Many thanks to the developers of Chart.js, and to the tutorial author at techrepublic.com
