Abstract Cybersecurity is an increasingly important challenge for computer systems. In this work, cyberattacks were modeled using an extension of the well-known Petri net formalism. That formalism, designated Petri nets with players, strategies, and costs, models the states of the cyberattack and events during the attack as markings and transition firings in the net respectively. The formalism models the attacker and defender as competing players who may observe the marking of a subset of the net and based on the observed marking act by changing the stochastic firing rates of a subset of the transitions in order to achieve their competing goals. Rate changes by the players incur a cost. Using the formalism, nets were constructed to model specific cyberattack patterns (cross-site scripting and spear phishing) documented in the Common Attack Pattern Enumeration and Classification database. The models were validated by a panel of cybersecurity experts in a structured face validation process. Given those validated nets, a reinforcement learning algorithm using an -Greedy policy was implemented and set to the task of learning which actions to take, i.e., which transition rates to change for the different observable markings, so as to accomplish the goals of the attacker or defender. Experiments were conducted with a dynamic (learning) attacker against a static (fixed) defender, a static attacker against a dynamic defender, and a dynamic attacker against a dynamic defender. In all cases, the reinforcement learning algorithm was able to improve its performance, in terms of achieving the player’s objective and reducing the cost of doing so, over time. These results demonstrate the potential of formally modeling cyberattacks and of applying reinforcement learning to improving cybersecurity.

中文翻译：

---

机器学习网络攻击和防御策略

摘要 网络安全是计算机系统面临的日益重要的挑战。在这项工作中，网络攻击是使用著名的 Petri 网形式主义的扩展来建模的。这种形式主义，用参与者、策略和成本指定 Petri 网，将网络攻击的状态和攻击期间的事件分别建模为网络中的标记和转换发射。形式主义将攻击者和防守者建模为竞争玩家，他们可以观察网子集的标记，并基于观察到的标记行为，通过改变转换子集的随机发射率来实现他们的竞争目标。玩家更改费率会产生成本。使用形式主义，网络被构建为对常见攻击模式枚举和分类数据库中记录的特定网络攻击模式（跨站点脚本和鱼叉式网络钓鱼）进行建模。这些模型由网络安全专家小组在结构化的面部验证过程中进行验证。给定这些经过验证的网络，实施了使用 -Greedy 策略的强化学习算法，并将其设置为学习采取哪些行动的任务，即，针对不同的可观察标记改变哪些转换率，以实现攻击者的目标或后卫。使用动态（学习）攻击者对抗静态（固定）防御者、静态攻击者对抗动态防御者以及动态攻击者对抗动态防御者进行实验。在所有情况下，随着时间的推移，强化学习算法能够提高其性能，实现玩家的目标并降低这样做的成本。这些结果证明了对网络攻击进行正式建模和应用强化学习来改善网络安全的潜力。