Currently, encryption (such as the Transport Layer Security protocol) is used by increasingly more network applications to protect their security and privacy, while it also benefits network attackers who can encrypt their traffic to evade detection. The detection of malicious encrypted traffic is becoming a critical task for cyber security. To accomplish this task, researchers have proposed several enlightening methods, including decryption followed by deep packet inspection (DPI), direct DPI on ciphertext and identification by machine learning algorithms. However, due to privacy violations or performance limitations, the state-of-the-art is far from satisfactory.

In this paper, we propose a novel framework and system called ME-Box (Machine learning and Evidence verification) for reliable detection of malicious encrypted traffic. ME-Box has middleboxes deployed in the network and agents installed on the sending hosts. Middleboxes first evaluate the trust degrees of encrypted flows by machine learning methods. If some flows are classified as suspicious, then middleboxes provide evidence of the evaluation results and request the corresponding session-keys from the agents. The agents verify the evidence, and if it is convincing, respond with the correct session-keys. With the session-keys, middleboxes finally decrypt the suspected encrypted flows and perform conventional DPI using intrusion signatures. We implement a prototype system of ME-Box and test it with real malware traffic. The experimental results show that ME-Box requires no modification of current cryptographic protocols and keeps end-users’ privacy well, and its performance is practically deployable.

当前，越来越多的网络应用程序使用加密（例如传输层安全性协议）来保护其安全性和私密性，同时也使可以加密其流量以逃避检测的网络攻击者受益。检测恶意加密流量已成为网络安全的关键任务。为了完成此任务，研究人员提出了多种启发性方法，包括解密后进行深度包检查（DPI），对密文进行直接DPI以及通过机器学习算法进行识别。但是，由于侵犯隐私或性能限制，最新技术远远不能令人满意。

在本文中，我们提出了一种新颖的框架和系统，称为ME-Box（机器学习和证据验证），用于可靠地检测恶意加密流量。ME-Box在网络中部署了中间盒，在发送主机上安装了代理。中间盒首先通过机器学习方法评估加密流的信任度。如果某些流量被归类为可疑，则中间框提供证据评估结果，并从代理请求相应的会话密钥。代理验证证据，如果有说服力，请使用正确的会话密钥进行响应。使用会话密钥，中间盒最终解密可疑的加密流，并使用入侵签名执行常规的DPI。我们实现了ME-Box的原型系统，并通过真实的恶意软件流量对其进行了测试。实验结果表明，ME-Box无需修改当前的加密协议，并且可以很好地保护最终用户的隐私，并且其性能实际上是可部署的。