Noise-resistant mechanisms for the detection of stealthy peer-to-peer botnets

Abstract

The problem of detection of malicious network traffic is adversarial in nature. Accurate detection of stealthy Peer-to-Peer botnets is an ongoing research problem. Past research on detection of P2P botnets has frequently used machine learning algorithms to build detection models. However, most prior work lacks the evaluation of such detection models in the presence of deliberate injection of noise by an adversary. Furthermore, detection of P2P botnets in the presence of benign P2P traffic has received little attention from the research community. This work proposes a novel approach for the detection of stealthy P2P botnets (in presence of benign P2P traffic) using conversation-based mechanisms and new features based on Fourier transforms and information entropy. We use real-world botnet data to compare the performance of our features with traditional 'flow-based' features employed by past research, and demonstrate that our approach is more resilient towards the injection of noise in the communication patterns by an adversary. We build detection models with multiple supervised machine learning algorithms. With our approach, we could detect P2P botnet traffic in the presence of injected noise with True Positive rate as high as 90%. (C) 2016 Elsevier B.V. All rights reserved.