Context

Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code.

Objective

This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software.

Method

We conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software.

Results

We found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them.

Conclusions

Scientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques.

中文翻译：

---

测试科学软件：系统文献综述。

语境

科学软件在关键决策中发挥着重要作用，例如根据气候模型进行天气预报，以及为研究出版物计算证据。最近，由于软件故障引起的错误，科学家不得不撤回出版物。系统测试可以识别代码中的此类错误。

客观的

本研究旨在确定测试科学软件时面临的具体挑战、提出的解决方案和未解决的问题。

方法

我们进行了系统的文献调查，以识别和分析相关文献。我们确定了 62 项提供有关测试科学软件的相关信息的研究。

结果

我们发现在测试科学软件时面临的挑战分为两大类：（1）由于科学软件的特性而出现的测试挑战，例如 oracle 问题；（2）由于科学家和软件工程师之间的文化差异而出现的测试挑战社区，例如将代码和它实现的模型视为不可分割的实体。此外，我们确定了可能克服这些挑战及其局限性的方法。最后，我们描述了未解决的挑战以及软件工程研究人员和从业人员如何帮助克服这些挑战。

结论

科学软件对测试提出了特殊的挑战。具体来说，科学家开发人员和软件工程师之间的文化差异，以及科学软件的特性，使测试变得更加困难。代码克隆检测等现有技术可以帮助改进测试过程。软件工程师在开发测试技术时应该考虑科学软件带来的特殊挑战，例如预言机问题。