The MUSICC project has created a proof-of-concept scenario database to be used as part of a type approval process for the verification of automated driving systems (ADS). This process must include a highly automated means of evaluating test results, as manual review at the scale required is impractical. This paper sets out a framework for assessing an ADS's behavioural safety in normal operation (i.e. performance of the dynamic driving task without component failures or malicious actions). Five top-level evaluation criteria for ADS performance are identified. Implementing these requires two types of outcome scoring rule: prescriptive (measurable rules which must always be followed) and risk-based (undesirable outcomes which must not occur too often). Scoring rules are defined in a programming language and will be stored as part of the scenario description. Risk-based rules cannot give a pass/fail decision from a single test case. Instead, a framework is defined to reach a decision for each functional scenario (set of test cases with common features). This considers statistical performance across many individual tests. Implications of this framework for hypothesis testing and scenario selection are identified.

中文翻译：

---

基于场景的自动驾驶系统测试的通过-失败标准

MUSICC 项目创建了一个概念验证场景数据库，用作验证自动驾驶系统 (ADS) 的型式批准流程的一部分。此过程必须包括评估测试结果的高度自动化方法，因为按所需规模进行手动审查是不切实际的。本文提出了一个评估 ADS 在正常操作中的行为安全性的框架（即在没有组件故障或恶意行为的情况下动态驾驶任务的性能）。确定了 ADS 性能的五个顶级评估标准。实施这些需要两种类型的结果评分规则：规定性（必须始终遵循的可衡量规则）和基于风险的（不得经常发生的不良结果）。评分规则是用编程语言定义的，并将作为场景描述的一部分进行存储。基于风险的规则不能从单个测试用例中给出通过/失败的决定。相反，定义了一个框架来为每个功能场景（具有共同特征的测试用例集）做出决定。这考虑了许多单独测试的统计性能。确定了该框架对假设检验和情景选择的影响。