Home » Uncategorized

Timestamp Data Visualization by Matplotlib

A large volume of timestamp data is a reality, this is common when we are dealing with networked devices. Typically a network of devices generate a large number of alerts. Mining of alert dataset provides insights about the network .

Recently, I came across a situation where a business user was looking for a multidimensional visualization of timestamp data.  Data was  about a network  of thousand plus devices and alarms  generated from the devices  about the status of the network  –  alarm transaction of few million records spread over six months and each record has a timestamp. His primary questions was :

1. Do we see a bunch of alarms during a specific time of the day.

2. Is this trend is consistent across months  and/or a pattern exists by the days of the month.

First step would be to create few visualization, then probably few measures around the questions to accept or reject hypotheses conjectured earlier. What he was looking for – a better description of the alarm patterns and linking with time, months and days.   As usual there could many approaches – we can have multiple ways of slicing the data and generate a large number of visualization etc.  

The aggregated data in this case – 4,320 records divided into 3 dimensions – month, day and hour.

The dataset in question can be structured  in two  ways –  1.  series of  4,320 data points  or 2. 2D Array of  4,230 cells. Without much details we can say  single bar and/or line makes a little sense here, if we make multiple bar or charts we may  miss the interrelation between the months, days and hours.

First solution was  a modified heat map as below:

2808324053

A  matrix of month-days X hour  with cell values being color coded as per the percentile buckets of the values. This picture provides a very high level view of the distribution of the events. In addition  from the picture we cannot make out  which is the exact time-hour when we the see the increased activity in the network. So  we can conclude that we need more granular view of the situation. 

I was looking for a better solution, finally with some secondary research, SO and others, developed a solution which can be thought as  visualization of data in circular coordinate system. 

2808324549

In the above diagram, a  single (thirty days) month has been represented by a collection of thirty concentric circles. Each circle has been divided into 24 arcs , with specific arc representing a particular time hour – the values of signals has been  color coded as per the specific scale. If we pick a specific disc and a specific arc it would tell us – distribution of signals for a 30 days period for that specific hour. In my opinion, this granular view is a really a good visualization for the questions above.

Matplotlib has good facility of plotting in polar coordinate system – demo page few examples. Plot above is a collection of line charts in polar coordinate system. For a specific data tuple – (day, hour and value), has been converted into – rθj, rθk ,… rθp, color-code ,subsequently, plotting  a line base on this conversion. This would be repeated for 4,320 data points.

Finally, polar coordinate based visualization could be useful  in specific  situations, for timestamp data it could all the more useful.  The original mathematical approach is here

Leave a Reply

Your email address will not be published. Required fields are marked *