IoT Anomaly detection - algorithms, techniques and open source implementation


Anomaly detection for IoT is one of the archetypal applications for IoT.

Anomaly detection techniques are also used outside of IoT.

In my teaching at the #universityofoxford - we use anomaly detection as a use case because it brings together many of the intricacies for IoT and also demonstrates the use of multiple #machinelearning and #deeplearning algorithms   

Long term, I am exploring the idea of creating an open source anomaly detector for IoT - both for my students and in general.

So, I  am exploring this  space from a research perspective and will share the ideas on Data Science Central.

I will continue to explore more and implement the solution as a series of use cases and strategies listed below with my students. If you are interested in contributing, please connect with me Ajit Jaokar - linkedin


Anomalies arise from a variety of cause including intrusion detection, fraud and data leakage

We could have a number of categories of anomalies – for example:

  • Point anomalies: specific data points that fall outside the norm
  • Contextual anomalies: if the data point is anomalous in a specific context – ex in a time window
  • Collective anomalies: where a collection of data points fall outside the norm

 A range of algorithmic approaches could be used in anomaly detection

  • Supervised learning: ex SVM or Deep neural networks
  • Semi-supervised Anomaly Detection: One-class SVMs and autoencoders – or density estimation approaches such as Gaussian Mixture approaches or Kernel Density Estimation – where we train the normal class and detect the anomaly as a deviation from the normal class.
  • Unsupervised anomaly detection: where we have no training data as such. Clustering is an example of unsupervised learning

IoT anomaly detection also encounters unbalanced datasets. So, we need approaches to handle unbalanced data such as resampling, under sampling and oversampling. Random over-sampling is also achieved by generating new synthetic data of minority class by interpolation, through techniques SMOTE and ADASYN.

Finally, a range of feature engineering techniques are used in anomaly detection which we list in the table below


We represent the analysis below (and the sources). 

Next steps - an open source IoT anomaly detector?

I will continue to explore more and implement the solution as a series of use cases and strategies listed below with my students. If you are interested in contributing, please connect with me Ajit Jaokar - linkedin

a sample of analysis below

you can download the full table from this link - IoT Anomaly Detection



Anomaly Type Method Features Application
[11] Collective Real-Valued Negative Selection self-organizing maps Network Traffic
[12] Internet Anomalies SVM  Genetic Algorithm for feature selection Real world NIDS
[13] Misuse Detection Bag of system calls Memory cells UNM System Call Sequence
[14] Collective CNN and RNN Deep CNN features Smart Home
[15] Contextual SVM - Human Activity
[16] Collective PCA and Fuzzy rule based Feature Reduction Smart Home
[17] Contextual SVM - Health care
[18] Collective Multi class SVM Deep maps Wearable gadget
[19] Collective Single Class SVM Binary representation Smart Home
[20] Collective Adversarial auto encoder (AAE) and variation auto encoder (VAE) Stripes and point anomalies Cifar 10
[21] Collective Decision Tree, Random Forest, and ANN Multi class features Network Traffic
[22] Point Random Forest machine Binary feature for point anomaly Network Traffic
[23] end-points despite IP spoofing network centric, behavior-learning based, anomaly detection approach behavior-learning based features Network Traffic 
[24] Point Un supervised learning - Aerospace
[25] Point Supervised on CTIV platform Deep Feature maps Railway track
[26] Contextual Supervised on historical data Random classifier for dimension reduction Road Track
[27] Contextual Supervised on historical data Random classifier for dimension reduction Air traffic control system
[28] Contextual Successive Cloud increasing complexity, and associate each model with a layer Detection  Delay
[29] SDN-Enable Ensemble learning deep auto-encoder to extract handy features benchmark datasets
[30] Point Long term threshold Analyze the fault data pattern Different Sensors
[31] Distributed attack Deep Model Hybrid counter parts Network Traffic
[32] Activity Attack Self-Learning device-type-specific communication  Network Traffic
[33] Collective NIDSs validate using TCP/IP Industrial IoT
[34] Equipment multi-stream CNN-based remote monitoring Shape  and texture irregularity Network Traffic
[35] Point Edge Computer Cloud none Air traffic control system
[36] Contextual LSTM Long-term data Features Smart Home
[37] Contextual deep learning-based method Deep Belief Network (DBN) a Feature maps Health Care
[38] Collective+ Point two-stage sliding Recurrent Auto encoder Raw time series Time Series
[39] Point Temporal Convolutional Network (HS-TCN) Stacking features IoT Communication
[40] Distributed Graph Neural Network multi-agent Features Network Traffic
[41] Distributed Neural Network Multi-layer ANN features Network Traffic
[42] Contextual SVM - Health care
[43] Contextual Random Forest machine Deep feature for  Contextual anomaly Network Traffic
[44] Contextual a cognitive-based middleware concealment Features Network Traffic
[45] Distributed Ultra-Lightweight Deep Packet Anomaly Detection bit pattern matching Network Appliances
[46] Distributed DNN a data-driven Feature selection Energy Management
[47] Distributed Unsupervised CNN auto-profiling Features Network Traffic
[48] Distributed multi-convolutional neural network (multi-CNN) Feature Fusion Network Traffic
[49] Distributed Development and Operations (DevOps) Method Rider Optimization Algorithm (ROA) Network Traffic
[50] Distributed Hierarchical clustering + LSTM M-estimator Network Traffic
[51] Collective Supervised Learning Anomalous Patterns  Smart Home
[52] Collective Supervised Learning Coxian duration Features Smart Kitchen
[53] Contextual Supervised Learning Hierarchical Markov features Smart Home
[54] Contextual Supervised Learning None Smart Home
[55] Contextual Supervised Learning Data source information Flight safety
[56] Contextual Un Supervised Learning Textual features Road Traffic
[57] Collective Supervised Learning Kernel Feature Space  Aerospace
[58] Collective Supervised Learning Probabilistic features Pump Trucks
[59] Collective Supervised Learning Statistical features Vehicle abnormality
[60] Point Supervised Learning None Trash Bin
[61] Point None low and high level reasoning Health Care
[62] Point Supervised Learning Single channel based features Health Care
[63] Contextual Supervised Learning EEG signal features Health Care
[64] Contextual Supervised Learning Features based on historic data Health Care
[65] Contextual Supervised Learning Person movement based features Health Care

References for papers used in above table

  1. Fahim, M., & Sillitti, A. (2019). Anomaly detection, analysis and prediction techniques in iot environment: A systematic literature review. IEEE Access7, 81664-81681.
  2. Behniafar, M., Nowroozi, A. R., & Shahriari, H. R. (2018). A Survey of Anomaly Detection Approaches in Internet of Things. ISeCure-The ISC International Journal of Information Security10(2), 79-92.
  3. Zarpelão, B. B., Miani, R. S., Kawakani, C. T., & de Alvarenga, S. C. (2017). A survey of intrusion detection in Internet of Things. Journal of Network and Computer Applications84, 25-37.
  4. Kim, A., Oh, J., Ryu, J., & Lee, K. (2020). A Review of Insider Threat Detection Approaches With IoT Perspective. IEEE Access8, 78847-78867.
  5. da Costa, K. A., Papa, J. P., Lisboa, C. O., Munoz, R., & de Albuquerque, V. H. C. (2019). Internet of Things: A survey on machine learning-based intrusion detection approaches. Computer Networks151, 147-157.
  6. Elrawy, M. F., Awad, A. I., & Hamed, H. F. (2018). Intrusion detection systems for IoT-based smart environments: a survey. Journal of Cloud Computing7(1), 21.
  7. Hassan, W. H. (2019). Current research on Internet of Things (IoT) security: A survey. Computer networks148, 283-294.
  8. Sharma, B., Sharma, L., & Lal, C. (2019, December). Anomaly Detection Techniques using Deep Learning in IoT: A Survey. In 2019 International Conference on Computational Intelligence and Knowledge Economy (ICCIKE)(pp. 146-149). IEEE.
  9. Singh, T., & Kumar, N. (2020). Machine learning models for intrusion detection in IoT environment: A comprehensive review. Computer Communications.
  10. Vishwakarma, R., & Jain, A. K. (2020). A survey of DDoS attacking techniques and defence mechanisms in the IoT network. Telecommunication Systems73(1), 3-25.
  11. González, F. A., & Dasgupta, D. (2003). Anomaly detection using real-valued negative selection. Genetic Programming and Evolvable Machines4(4), 383-403.
  12. Shon, T., Kim, Y., Lee, C., & Moon, J. (2005, June). A machine learning framework for network anomaly detection using SVM and GA. In Proceedings from the sixth annual IEEE SMC information assurance workshop(pp. 176-183). IEEE.
  13. Kang, D. K., Fuller, D., & Honavar, V. (2005, June). Learning classifiers for misuse and anomaly detection using a bag of system calls representation. In Proceedings from the Sixth Annual IEEE SMC Information Assurance Workshop(pp. 118-125). IEEE.
  14. Han, N., Gao, S., Li, J., Zhang, X., & Guo, J. (2018, August). Anomaly detection in health data based on deep learning. In 2018 International Conference on Network Infrastructure and Digital Content (IC-NIDC)(pp. 188-192). IEEE.
  15. Yin, J., Yang, Q., & Pan, J. J. (2008). Sensor-based abnormal human-activity detection. IEEE Transactions on Knowledge and Data Engineering20(8), 1082-1090.
  16. Mahmoud, S. M., Lotfi, A., & Langensiepen, C. (2012, June). User activities outlier detection system using principal component analysis and fuzzy rule-based system. In Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments(pp. 1-8).
  17. Shin, J. H., Lee, B., & Park, K. S. (2011). Detection of abnormal living patterns for elderly living alone using support vector data description. IEEE Transactions on Information Technology in Biomedicine15(3), 438-448.
  18. Palaniappan, A., Bhargavi, R., & Vaidehi, V. (2012, April). Abnormal human activity recognition using SVM based approach. In 2012 International Conference on Recent Trends in Information Technology(pp. 97-102). IEEE.
  19. Jakkula, V., & Cook, D. (2011, August). Detecting anomalous sensor events in smart home data for enhancing the living experience. In Workshops at the twenty-fifth AAAI conference on artificial intelligence.
  20. Chalapathy, R., Toth, E., & Chawla, S. (2018, September). Group anomaly detection using deep generative models. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases(pp. 173-189). Springer, Cham.
  21. Hasan, M., Islam, M. M., Zarif, M. I. I., & Hashem, M. M. A. (2019). Attack and anomaly detection in IoT sensors in IoT sites using machine learning approaches. Internet of Things7, 100059.
  22. Alrashdi, I., Alqazzaz, A., Aloufi, E., Alharthi, R., Zohdy, M., & Ming, H. (2019, January). Ad-iot: Anomaly detection of iot cyberattacks in smart city using machine learning. In 2019 IEEE 9th Annual Computing and Communication Workshop and Conference (CCWC)(pp. 0305-0310). IEEE.
  23. Bhatia, R., Benno, S., Esteban, J., Lakshman, T. V., & Grogan, J. (2019, December). Unsupervised machine learning for network-centric anomaly detection in IoT. In Proceedings of the 3rd ACM CoNEXT Workshop on Big DAta, Machine Learning and Artificial Intelligence for Data Communication Networks(pp. 42-48).
  24. Bai, M., Liu, J., Chai, J., Zhao, X., & Yu, D. (2020). Anomaly detection of gas turbines based on normal pattern extraction. Applied Thermal Engineering166, 114664.
  25. Gibert, X., Patel, V. M., & Chellappa, R. (2016). Deep multitask learning for railway track inspection. IEEE transactions on intelligent transportation systems18(1), 153-164.
  26. Bose, B., Dutta, J., Ghosh, S., Pramanick, P., & Roy, S. (2018, February). D&RSense: Detection of driving patterns and road anomalies. In 2018 3rd International Conference On Internet of Things: Smart Innovation and Usages (IoT-SIU)(pp. 1-7). IEEE.
  27. Farshchi, M., Weber, I., Della Corte, R., Pecchia, A., Cinque, M., Schneider, J. G., & Grundy, J. (2018, September). Contextual anomaly detection for a critical industrial system based on logs and metrics. In 2018 14th European Dependable Computing Conference (EDCC)(pp. 140-143). IEEE.
  28. Ngo, M. V., Luo, T., Chaouchi, H., & Quek, T. Q. (2020). Contextual-Bandit Anomaly Detection for IoT Data in Distributed Hierarchical Edge Computing. arXiv preprint arXiv:2004.06896.
  29. Tsogbaatar, E., Bhuyan, M. H., Taenaka, Y., Fall, D., Gonchigsumlaa, K., Elmroth, E., & Kadobayashi, Y. (2020, June). SDN-Enabled IoT Anomaly Detection Using Ensemble Learning. In IFIP International Conference on Artificial Intelligence Applications and Innovations(pp. 268-280). Springer, Cham.
  30. Tsai, F. K., Chen, C. C., Chen, T. F., & Lin, T. J. (2019, April). Sensor Abnormal Detection and Recovery Using Machine Learning for IoT Sensing Systems. In 2019 IEEE 6th International Conference on Industrial Engineering and Applications (ICIEA)(pp. 501-505). IEEE.
  31. Diro, A. A., & Chilamkurti, N. (2018). Distributed attack detection scheme using deep learning approach for Internet of Things. Future Generation Computer Systems82, 761-768.
  32. Nguyen, T. D., Marchal, S., Miettinen, M., Fereidooni, H., Asokan, N., & Sadeghi, A. R. (2019, July). DÏoT: A federated self-learning anomaly detection system for IoT. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS)(pp. 756-767). IEEE.
  33. Muna, A. H., Moustafa, N., & Sitnikova, E. (2018). Identification of malicious activities in industrial internet of things based on deep learning models. Journal of Information Security and Applications41, 1-11.
  34. Hou, R., Pan, M., Zhao, Y., & Yang, Y. (2019). Image anomaly detection for IoT equipment based on deep learning. Journal of Visual Communication and Image Representation64, 102599.
  35. Ferrari, P., Rinaldi, S., Sisinni, E., Colombo, F., Ghelfi, F., Maffei, D., & Malara, M. (2019, June). Performance evaluation of full-cloud and edge-cloud architectures for Industrial IoT anomaly detection based on deep learning. In 2019 II Workshop on Metrology for Industry 4.0 and IoT (MetroInd4. 0&IoT)(pp. 420-425). IEEE.
  36. Utomo, D., & Hsiung, P. A. (2019, May). Anomaly Detection at the IoT Edge using Deep Learning. In 2019 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW)(pp. 1-2). IEEE.
  37. Manimurugan, S., Al-Mutairi, S., Aborokbah, M. M., Chilamkurti, N., Ganesan, S., & Patan, R. (2020). Effective Attack Detection in Internet of Medical Things Smart Environment Using a Deep Belief Neural Network. IEEE Access8, 77396-77404.
  38. Yin, C., Zhang, S., Wang, J., & Xiong, N. N. (2020). Anomaly Detection Based on Convolutional Recurrent Autoencoder for IoT Time Series. IEEE Transactions on Systems, Man, and Cybernetics: Systems.
  39. Cheng, Y., Xu, Y., Zhong, H., & Liu, Y. (2020). Leveraging Semi-supervised Hierarchical Stacking Temporal Convolutional Network for Anomaly Detection in IoT Communication. IEEE Internet of Things Journal.
  40. Protogerou, A., Papadopoulos, S., Drosou, A., Tzovaras, D., & Refanidis, I. (2020). A graph neural network method for distributed anomaly detection in IoT. Evolving Systems, 1-18.
  41. Hodo, E., Bellekens, X., Hamilton, A., Dubouilh, P. L., Iorkyase, E., Tachtatzis, C., & Atkinson, R. (2016, May). Threat analysis of IoT networks using artificial neural network intrusion detection system. In 2016 International Symposium on Networks, Computers and Communications (ISNCC)(pp. 1-6). IEEE.
  42. Ukil, A., Bandyoapdhyay, S., Puri, C., & Pal, A. (2016, March). IoT healthcare analytics: The importance of anomaly detection. In 2016 IEEE 30th international conference on advanced information networking and applications (AINA)(pp. 994-997). IEEE.
  43. Tama, B. A., & Rhee, K. H. (2017). Attack classification analysis of IoT network via deep learning approach.  Briefs Inf. Commun. Technol. Evol.(ReBICTE)3, 1-9.
  44. Elmisery, A. M., Sertovic, M., & Gupta, B. B. (2017). Cognitive privacy middleware for deep learning mashup in environmental IoT. IEEE access6, 8029-8041.
  45. Summerville, D. H., Zach, K. M., & Chen, Y. (2015, December). Ultra-lightweight deep packet anomaly detection for Internet of Things devices. In 2015 IEEE 34th international performance computing and communications conference (IPCCC)(pp. 1-8). IEEE.
  46. Samani, E., Khaledian, P., Aligholian, A., Papalexakis, E., Cun, S., Nazari, M. H., & Mohsenian-Rad, H. (2020, February). Anomaly detection in iot-based pir occupancy sensors to improve building energy efficiency. In 2020 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT)(pp. 1-5). IEEE.
  47. Hwang, R. H., Peng, M. C., Huang, C. W., Lin, P. C., & Nguyen, V. L. (2020). An Unsupervised Deep Learning Model for Early Network Traffic Anomaly Detection. IEEE Access8, 30387-30399.
  48. Li, Y., Xu, Y., Liu, Z., Hou, H., Zheng, Y., Xin, Y., ... & Cui, L. (2020). Robust detection for network intrusion of industrial IoT based on multi-CNN fusion. Measurement154, 107450.
  49. Sarma, S. K. (2020, May). Rider Optimization based Optimized Deep-CNN towards Attack Detection in IoT. In 2020 4th International Conference on Intelligent Computing and Control Systems (ICICCS)(pp. 163-169). IEEE.
  50. Shukla, R. M., & Sengupta, S. (2020). Scalable and Robust Outlier Detector using Hierarchical Clustering and Long Short-Term Memory (LSTM) Neural Network for the Internet of Things. Internet of Things, 100167.
  51. Ordóñez, F. J., de Toledo, P., & Sanchis, A. (2015). Sensor-based Bayesian detection of anomalous living patterns in a home setting. Personal and Ubiquitous Computing19(2), 259-270.
  52. Duong, T. V., Bui, H. H., Phung, D. Q., & Venkatesh, S. (2005, June). Activity recognition and abnormality detection with the switching hidden semi-markov model. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)(Vol. 1, pp. 838-845). IEEE.
  53. Kang, W., Shin, D., & Shin, D. (2010, October). Detecting and predicting of abnormal behavior using hierarchical markov model in smart home network. In 2010 IEEE 17Th International Conference on Industrial Engineering and Engineering Management(pp. 410-414). IEEE.
  54. Cuddihy, P., Weisenberg, J., Graichen, C., & Ganesh, M. (2007, June). Algorithm to automatically detect abnormally long periods of inactivity in a home. In Proceedings of the 1st ACM SIGMOBILE international workshop on Systems and networking support for healthcare and assisted living environments(pp. 89-94).
  55. He, Y., Peng, Y., Wang, S., Liu, D., & Leong, P. H. (2017). A structured sparse subspace learning algorithm for anomaly detection in UAV flight data. IEEE Transactions on Instrumentation and Measurement67(1), 90-100.
  56. Rodrigues, D. O., Santos, F. A., Akabane, A. T., Cabral, R., Immich, R., Junior, W. L., ... & Cerqueira, E. (2019). Computa\c {c}\~ ao Urbana da Teoria\a Pr\'atica: Fundamentos, Aplica\c {c}\~ oes e Desafios. arXiv preprint arXiv:1912.05662.
  57. Fujimaki, R., Yairi, T., & Machida, K. (2005, August). An approach to spacecraft anomaly detection problem using kernel feature space. In Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining(pp. 401-410).
  58. Ding, J., Liu, Y., Zhang, L., Wang, J., & Liu, Y. (2016). An anomaly detection approach for multiple monitoring data series based on latent correlation probabilistic model. Applied Intelligence44(2), 340-361.
  59. Han, M. L., Lee, J., Kang, A. R., Kang, S., Park, J. K., & Kim, H. K. (2015, December). A statistical-based anomaly detection method for connected cars in internet of things environment. In International Conference on Internet of Vehicles(pp. 89-97). Springer, Cham.
  60. Amores, J., Maes, P., & Paradiso, J. (2015, September). Bin-ary: detecting the state of organic trash to prevent insalubrity. In Adjunct Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2015 ACM International Symposium on Wearable Computers(pp. 313-316).
  61. Pham, T. T., Nguyen, D. N., Dutkiewicz, E., McEwan, A. L., & Leong, P. H. (2017, May). Wearable healthcare systems: A single channel accelerometer based anomaly detector for studies of gait freezing in Parkinson's disease. In 2017 IEEE International Conference on Communications (ICC)(pp. 1-5). IEEE.
  62. Tonchev, K., Koleva, P., Manolova, A., Tsenov, G., & Poulkov, V. (2016, June). Non-intrusive sleep analyzer for real time detection of sleep anomalies. In 2016 39th International Conference on Telecommunications and Signal Processing (TSP)(pp. 400-404). IEEE.
  63. Puri, C., Ukil, A., Bandyopadhyay, S., Singh, R., Pal, A., & Mandana, K. (2016, June). iCarMa: Inexpensive Cardiac Arrhythmia Management--An IoT Healthcare Analytics Solution. In Proceedings of the first workshop on IoT-enabled healthcare and wellness technologies and systems(pp. 3-8).
  64. Zhu, Y. (2011). Automatic detection of anomalies in blood glucose using a machine learning approach. Journal of Communications and Networks13(2), 125-131.
  65. Burchfield, T. R., & Venkatesan, S. (2007, June). Accelerometer-based human abnormal movement detection in wireless sensor networks. In Proceedings of the 1st ACM SIGMOBILE international workshop on Systems and networking support for healthcare and assisted living environments(pp. 67-69).


References used in this article

Learning from imbalanced data: open challenges and future directions

Anomaly Detection Strategies for IoT Sensors


 Image source: mres.uni-postsdam


Views: 2321

Tags: dsc_analytics, dsc_iot, dsc_tagged


You need to be a member of Data Science Central to add comments!

Join Data Science Central

Comment by Fernando Agustin Méndez Monroy on July 3, 2020 at 5:59am

Sometimes we want to get rid of anomalies, sometimes we actually want to understand them. One example is Climate Variability due to Climate Change.

© 2021   TechTarget, Inc.   Powered by

Badges  |  Report an Issue  |  Privacy Policy  |  Terms of Service