Smart IDS: Reducing False Alarms in Real-Time and Live Network Intrusion Detection Using Machine Learning

Abstract: Smart Intrusion Detection System (IDS) plays a vital role in recognizing network malicious activities in the present network environment. The traditional signature-based methods are usually ineffective towards detecting zero- day and new cyber threats. This paper has presented SMART IDS, as a machine learning based intrusion detection system that attempts to minimize the false alarms and preserve the detection accuracy of the system in both simulated and real network environments. The system is running in simulation mode whereby the CICIDS2017 dataset is used to train and test the model and live capture mode whereby real time packet analysis is performed. An extensive preprocessing pipeline is used. This involves cleaning of data, selection of features, encoding and standardization to provide an assuring model performance. A variety of machine learning models are utilized and considered: Decision Tree, Random Forest, XGBoost, and Logistic Regression. To improve the detection capability, a custom weighted ensemble method is suggested. It has an attack override feature, which puts the emphasis on threat detection and minimizes false negatives. In the experimental area, the accuracy of the Random Forest model is high (99.86). At the same time, the ensemble approach enhances recall and reduces the number of missed attacks as compared to the single models. Real-time packet capture, which is added with the help of PyShark, and deployment with a Flask-based API make the system applicable to dynamic network environments. The SMART IDS is a scaled, adaptable, and efficient framework, which could be applied to handle the cybersecurity reality in contemporary times. It uses machine learning methods with real-time analysis, which greatly improves the performance of intrusion detection and decreases the number of false alarms.

Keywords: Cybersecurity, Intrusion Detection System, Machine Learning, Network Security, Real-Time Detection