Practical task allocation for software fault-tolerance and its implementation in embedded automotive systems

Abstract

Due to the advent of active safety features and automated driving capabilities, the complexity of embedded computing systems within automobiles continues to increase. Such advanced driver assistance systems (ADAS) are inherently safety-critical and must tolerate failures in any subsystem. However, fault-tolerance in safety-critical systems has been traditionally supported by hardware replication, which is prohibitively expensive in terms of cost, weight, and size for the automotive market. Recent work has studied the use of software-based fault-tolerance techniques that utilize task-level hot and cold standbys to tolerate fail-stop processor and task failures. The benefit of using standbys is maximal when a task and any of its standbys obey the placement constraint of not being co-located on the same processor. We propose a new heuristic based on a “tiered” placement constraint, and show that our heuristic produces a better task assignment that saves at least one processor up to 40% of the time relative to the best known heuristic to date. We then introduce a task allocation algorithm that, for the first time to our knowledge, leverages the run-time attributes of cold standbys. Our empirical study finds that our heuristic uses no more than one additional processor in most cases relative to an optimal allocation that we construct for evaluation purposes using a creative technique. We also extend our heuristic to support mixed-criticality systems which allow for overload operation. We have designed and implemented our software fault-tolerance framework in AUTOSAR, an automotive industry standard. We use this implementation to provide an experimental evaluation of our task-level fault-tolerance features. Finally, we present an analysis of the worst-case behavior of our task recovery features.

Appendices

Appendix

The BFD-P and R-BFD Heuristics

In this section, we provide a brief overview of the BFD-P and R-BFD heuristics (Kim et al. 2010).

The BFD-P algorithm follows the below steps,

• (1) Sort tasks including replicas in the decreasing order of utilization.

• (2) Fit every task into the best fit processor obeying the placement constraint, i.e., any task should not be co-located with its replica.

• (3) Add a new processor if a task does not fit any bin.

• (4) Iterate until no tasks remain.

The R-BFD algorithm follows the below steps,

• (1) The given tasks are sorted in decreasing order of utilization.

• (2) The primary tasks are extracted and allocated first using the BFD-P heuristic.

• (3) The replicas are then allocated one by one, highest order replicas first, i.e., opposite to the TPCD approach.

• (4) Add a new processor if a task does not fit any bin.

• (5) Iterate until no tasks remain.

1.1 TPCD primary redistribution

As highlighted in Sect. 4.1 the TPCD algorithm can result in an allocation where a processor becomes dominant and runs only primaries. This can be seen in the allocation produced by the example in Sect. 4.1. Figure 22 represents the TPCD allocation produced with backup types annotated. As can be seen the last processor runs all primaries and becomes the dominant processor. This is not ideal in a safety-critical systems. This type of allocation is a result of the fact that TPCD allocates tasks in tiers. This same allocation scheme also allows backups to be clubbed together. If the backups of some of the primaries on the dominant processor run identical copies as the primary it is possible to swap their positions with the primary to produce a more balanced distribution of primaries and standbys. This is highlighted in Fig. 23.

Fig. 22

TPCD solution to 2a with backup types highlighted (Color figure online)

Fig. 23

TPCD solution to 2a with primary redistribution [(P primary, B backup (color online)]

1.2 Group maintenance

In this section, we describe some of the messages that are passed between the primary and its backups to maintain group membership. The primary acts as the leader of the group and has information about all its members. It shares this information with its group members so that, in the case of a primary failure, the remaining group members and the new leader can continue to maintain the group.

Since the task to node assignment is done beforehand, each task is pre-defined to run as either a primary, hot or cold standby. On startup, all nodes enter a group formation phase. In this phase, a primary periodically broadcasts a $Group_{create}$ message along with its heartbeat. It then waits for a pre-configured interval $T_{timeout_{pri}}$ (typically a multiple of its period $T_{pri}$) to receive any $Group_{created}$ responses, which would indicate that a primary already exists. If there is no such response in this interval, the primary moves out of the startup phase and enters normal operation, where it starts producing application outputs, listening for heartbeats from its standbys and producing heartbeats, state and group information (i.e. the normal life cycle outlined in Sect. 3.2).

Each standby on startup periodically produces a $Group_{join}$ message. On receiving the $Group_{create}$ or $Group_{created}$ message from the primary, it transitions out of the startup phase and starts its normal operation and producing heartbeat messages. If a standby does not receive a $Group_{create}$ message for a period of $T_{timeout_{sb}}$ units, then it declares primary failure and transitions to the normal mode of operation as the new primary. Figure 17a represents the final stable state of the system after the groups are created and also shows the messages exchanged within the group during normal operation.

If a primary fails and is later restarted, it will broadcast a $Group_{create}$ message along with its heartbeat. This time, the standby will have taken over as the new primary and it will respond to the $Group_{create}$ message with a $Group_{created}$ message indicating to the re-launched primary that it should run as a standby. Figure 17c represents the final stable state of the system after such a dynamic reconfiguration.