Subscribe to DSC Newsletter

Applying Face Emotion Recognition API Technology to Video of Nawaz Sharif's Address to Nation after Panama Papers

Machine Learning is being applied to almost everything these days, and the results are immaculate. With the introduction of Machine Learning APIs, developers don't have to train their own Machine Learning Algorithms, rather they can use these Machine Learning API's to create most interesting applications.

Recently, I red an article in which Ben Hubl applied Microsoft Cognitive Service Emotions API to do emotions analysis of video of Hillary and Trumps last debate. I was amazed to see the accuracy of results, and the way graphs of emotions were shown per frame of the video.

This inspired me to do video emotions analysis of a Prime Minister Nawaz Sharif address to the nation after Panama Papers were published. I wanted to check what kind of emotions he showed during the speech, and the interesting thing is that emotions can be tracked for the whole video i.e. how the emotions change during the address.

The picture above shows Emotions Analysis of a picture from the address. Microsoft Cognitive Services Emotions API gives percentages of what it thinks the emotion is based on the picture. In the above picture, the highest number is for Neutral with 0.42, followed by Anger and Disgust with 0.30 and 0.20 respectively.

This was emotions analysis of a picture from the Prime Minister's address. Now coming to the emotions analysis of the whole video, it tracks the emotions during the whole video and provides a JSON(Java Script Object Notation) file which reports emotions per frame. I used Python to get the results from Microsoft Cognitive Video Emotions API, and then used Python's matplotlib and R's ggplot 2 to make visualizations. I will share the detailed code on my GitHub.

After getting the results for the whole video, I took mean of all the emotions and made a Bar Chart using Python's matplotlib library. Here is the Bar Chart.

As expected, Anger has the highest mean, followed by sadness and disgust, which means that Prime Minister was quite angry during the speech. Surprise and contempt come on the lower side along with fear and happiness which are almost non existent emotions during the speech.

So emotions tell us a lot about what kind of speech it was, and the emotions which the Prime Minister went through during the speech. Histogram showed the average emotions during the speech, but we also have the emotion per frame, so we can visualize it on a time line i.e. Frame on x axis and the value of emotion of the y axis, and thereby see the change in emotions during the whole speech. Here is that graph which I made using ggplot 2 visualization library of R.

Here we can see that speech started with high anger, sadness and disgust, which then dipped and smoothed off. Prime Minister started with a low value of happiness, but it latter improved and at the end of the speech happiness took over disgust. The remaining emotions like fear, contempt and surprise remained low during the whole speech.

This kind of analysis can be supplemented with natural language processing i.e. concerned with the interactions between computers and human (natural) languages. We can check which words are associated with which emotions, which I intend to do in the future. Any guess which emotion of the Prime Minister could be associated with the word Imran? lol

I have used smoothed conditional means in the visualization above using gglpot 2 geom_smooth() using method = 'loess'. This is for the more technical people, I will share the full code on my GitHub soon.

I wrote this article to show that we have entered a new era, and how Machine Learning is impacting every industry. The traditional print and digital media can use Machine Learning to get insights like the one I have shown from videos and pictures on a massive scale in real time.

Views: 10124

Comment

You need to be a member of Data Science Central to add comments!

Join Data Science Central

Comment by Ignacio Perez on May 23, 2017 at 3:34pm

Interesting use of the technolgy. Would be great if you share your coding on your GitHub

Comment by Syed Mudassir Hussain on May 14, 2017 at 6:12am

An interesting application of the technology. I am sure, the graph lines can be edited easily by the author and updated. However, there is a lot more to learn by using this technology.

Comment by Michał Witkowski on May 13, 2017 at 12:16am

That's such a great idea O:

Not too fortunate selection of colours for the frame x emotions graph, though - very difficult to differentiate between fear and happiness.

Follow Us

Videos

  • Add Videos
  • View All

Resources

© 2017   Data Science Central   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service