Since the inception of Andoroid OS, smartphones sales have been growing exponentially, and today it enjoys the monopoly in the smartphone marketplace. The widespread adoption of Android smartphones has drawn the attention of malware designers, which threatens the Android ecosystem. The current state-of-the-art Android malware detection systems are based on machine learning and deep learning models. Despite having superior performance, these models are susceptible to adversarial attack. Therefore in this paper, we developed eight Android malware detection models based on machine learning and deep neural network and investigated their robustness against the adversarial attacks. For the purpose, we created new variants of malware using Reinforcement Learning, which will be misclassified as benign by the existing Android malware detection models. We propose two novel attack strategies, namely single policy attack and multiple policy attack using reinforcement learning for white-box and grey-box scenario respectively. Putting ourselves in adversary’ shoes, we designed adversarial attacks on the detection models with the goal of maximising fooling rate, while making minimum modifications to the Android application and ensuring that the app’s functionality and behaviour does not change. We achieved an average fooling rate of 44.21% and 53.20% across all the eight detection models with maximum five modifications using a single policy attack and multiple policy attack, respectively. The highest fooling rate of 86.09% with five changes was attained against the decision tree based model using the multiple policy approach. Finally, we propose an adversarial defence strategy which reduces the average fooling rate by threefold to 15.22% against a single policy attack, thereby increasing the robustness of the detection models i.e. the proposed model can effectively detect variants (metamorphic) of malware. The experimental analysis shows that our proposed Android malware detection system using reinforcement learning is more robust against adversarial attacks.

自从Andoroid OS诞生以来，智能手机的销售量呈指数增长，如今，它在智能手机市场上享有垄断地位。Android智能手机的广泛采用引起了恶意软件设计者的关注，这威胁了Android生态系统。当前最新的Android恶意软件检测系统基于机器学习和深度学习模型。尽管具有出色的性能，但这些模型容易受到对抗性攻击。因此，在本文中，我们基于机器学习和深度神经网络开发了8个Android恶意软件检测模型，并研究了它们对对抗攻击的鲁棒性。为此，我们使用Reinforcement Learning创建了新的恶意软件变种，现有的Android恶意软件检测模型会将其错误分类为良性。我们提出了两种新颖的攻击策略，分别是针对白盒和灰盒场景使用强化学习的单策略攻击和多策略攻击。让自己陷入对手的困境，我们对检测模型进行了对抗攻击，目的是最大程度地提高愚弄率，同时对Android应用程序进行最少的修改，并确保该应用程序的功能和行为不会改变。在所有八个检测模型中，使用一次策略攻击和多次策略攻击最多进行五次修改，我们的平均欺骗率达到了44.21％和53.20％。使用多策略方法，基于决策树的模型在五次更改后的最高愚弄率为86.09％。最后，我们提出了一种对抗策略，可以将针对单一策略攻击的平均愚弄率降低三倍，达到15.22％，从而提高检测模型的鲁棒性，即所提出的模型可以有效地检测恶意软件的变体（变形）。实验分析表明，我们提出的使用强化学习的Android恶意软件检测系统对于对抗攻击更具鲁棒性。