Continuous experimentation and the cyber–physical systems challenge: An overview of the literature and the industrial perspective

Highlights

• Continuous Experimentation on cyber–physical systems is desirable but challenged.

• The state-of-the-art focuses more on the challenges than on solution proposals.

• The majority of literature comprises conceptual studies and empirical investigations.

• A solid state-of-practice has not been achieved yet.

Abstract

Context:

New software development patterns are emerging aiming at accelerating the process of delivering value. One is Continuous Experimentation, which allows to systematically deploy and run instrumented software variants during development phase in order to collect data from the field of application. While currently this practice is used on a daily basis on web-based systems, technical difficulties challenge its adoption in fields where computational resources are constrained, e.g., cyber–physical systems and the automotive industry.

Objective:

This paper aims at providing an overview of the engagement on the Continuous Experimentation practice in the context of cyber–physical systems.

Method:

A systematic literature review has been conducted to investigate the link between the practice and the field of application. Additionally, an industrial multiple case study is reported.

Results:

The study presents the current state-of-the-art regarding Continuous Experimentation in the field of cyber–physical systems. The current perspective of Continuous Experimentation in industry is also reported.

Conclusions:

The field has not reached maturity yet. More conceptual analyses are found than solution proposals and the state-of-practice is yet to be achieved. However it is expected that in time an increasing number of solutions will be proposed and validated.