Context The testing of software-intensive systems is performed in different test stages each having a large number of test cases. These test cases are commonly derived from requirements. Each test stages exhibits specific demands and constraints with respect to their degree of detail and what can be tested. Therefore, specific test suites are defined for each test stage. In this paper, the focus is on the domain of embedded systems, where, among others, typical test stages are Software- and Hardware-in-the-loop. Objective Monitoring and controlling which requirements are verified in which detail and in which test stage is a challenge for engineers. However, this information is necessary to assure a certain test coverage, to minimize redundant testing procedures, and to avoid inconsistencies between test stages. In addition, engineers are reluctant to state their requirements in terms of structured languages or models that would facilitate the relation of requirements to test executions. Method With our approach, we close the gap between requirements specifications and test executions. Previously, we have proposed a lightweight markup language for requirements which provides a set of annotations that can be applied to natural language requirements. The annotations are mapped to events and signals in test executions. As a result, meaningful insights from a set of test executions can be directly related to artifacts in the requirements specification. In this paper, we use the markup language to compare different test stages with one another. Results We annotate 443 natural language requirements of a driver assistance system with the means of our lightweight markup language. The annotations are then linked to 1300 test executions from a simulation environment and 53 test executions from test drives with human drivers. Based on the annotations, we are able to analyze how similar the test stages are and how well test stages and test cases are aligned with the requirements. Further, we highlight the general applicability of our approach through this extensive experimental evaluation. Conclusion With our approach, the results of several test levels are linked to the requirements and enable the evaluation of complex test executions. By this means, practitioners can easily evaluate how well a systems performs with regards to its specification and, additionally, can reason about the expressiveness of the applied test stage.

中文翻译：

---

我在测试什么？在哪里测试？比较基于轻量级需求注释的测试程序

上下文 软件密集型系统的测试在不同的测试阶段进行，每个阶段都有大量的测试用例。这些测试用例通常来自需求。每个测试阶段都表现出关于它们的详细程度和可以测试的内容的特定要求和限制。因此，为每个测试阶段定义了特定的测试套件。在本文中，重点是嵌入式系统领域，其中典型的测试阶段是软件和硬件在环。目标 监控和控制哪些要求在哪些细节以及在哪个测试阶段得到验证对工程师来说是一个挑战。然而，这些信息对于确保一定的测试覆盖率、最小化冗余测试程序以及避免测试阶段之间的不一致是必要的。此外，工程师不愿意用结构化语言或模型来陈述他们的需求，这些语言或模型会促进需求与测试执行的关系。方法 通过我们的方法，我们缩小了需求规范和测试执行之间的差距。之前，我们为需求提出了一种轻量级的标记语言，它提供了一组可以应用于自然语言需求的注释。注释被映射到测试执行中的事件和信号。因此，来自一组测试执行的有意义的见解可以与需求规范中的工件直接相关。在本文中，我们使用标记语言来比较不同的测试阶段。结果我们使用我们的轻量级标记语言注释了驾驶员辅助系统的 443 种自然语言要求。然后将注释链接到来自模拟环境的 1300 个测试执行和来自人类驾驶员试驾的 53 个测试执行。基于注释，我们能够分析测试阶段的相似程度以及测试阶段和测试用例与需求的一致性。此外，我们通过这种广泛的实验评估强调了我们方法的普遍适用性。结论 使用我们的方法，多个测试级别的结果与需求相关联，并能够评估复杂的测试执行。通过这种方式，从业者可以轻松地评估系统在其规范方面的表现，此外，