Zero-Day Attack Detection Using Autoencoder and XGBoost

Abstract

Advances in information and communication technology have significantly impacted progress in various sectors, but they have also given rise to increasingly complex network security threats. Cyberattacks such as Distributed Denial of Service (DDoS), ransomware, and software vulnerability exploits continue to increase year after year. Signature-based Intrusion Detection Systems are often ineffective in identifying novel cyberattacks since they rely solely on previously known attack patterns. To address this limitation, this study proposes a hybrid approach that integrates Autoencoders, including Dense and Memory-Augmented variants, with Extreme Gradient Boosting (XGBoost) to enhance zero-day attack detection using the UNSW-NB15 dataset. The research methodology encompasses data exploration, preprocessing with a split-before-transform strategy to prevent information leakage, Autoencoder training to model normal network behavior, reconstruction error computation for anomaly detection under both fixed and adaptive thresholding, and the utilization of these errors as input features for XGBoost classification. Experimental results demonstrate that adaptive thresholding improves F1 performance compared to fixed thresholds, while the hybrid Autoencoder–XGBoost integration achieves a significant performance boost. The proposed model consistently obtained F1 scores above 0.80 and PR-AUC values exceeding 0.81 with a balanced trade-off between precision and recall. These findings confirm that the hybrid approach is more effective, consistent, and adaptive in detecting intrusions, particularly zero-day attacks, and highlight its potential as a robust framework for advancing network security in dynamic threat environments.