Scenario-Based Testing of Cyber-Physical Systems

Description

In recent years, the development of automated and autonomously operating cyber-physical systems such as cars or drones is on the rise. Testing the safe behavior of these complex systems in all relevant situations is a challenge that is not easily solved.
Due to the high amount of needed test samples, the verification and validation of cyber-physical systems only with real-world testing is infeasible. To complement these efforts, we propose the use of scenario-based testing. In this method, the cyber-physical system is tested in different scenario types, which describe situations in which we aim to test the system's behavior. For cars, examples of these scenario types are "Lane Change" or "Cut-In". Each of these scenario types includes parameters to describe its various aspects, e.g., starting position of the system under test. When generating test cases for these scenario types, concrete values are assigned to all parameters of the considered scenario type.
 

Since it is not feasible to test the cyber-physical system in all possible scenario instances of one scenario type, we need to select "good" test cases that challenge the system under test. A test case is a "good" test case if it can reveal potential faults in the system. When generating these "good" test cases, different optimization techniques such as search-based techniques can be used.

Publications

Florian Hauer, Bernd Holzmüller: A sound approach to scenario-based testing as the basis for safety argumentations, Automotive Testing Technology Magazine, Special Issue, September 2020, page 93

Florian Hauer; Alexander Pretschner; Bernd Holzmüller: Re-Using Concrete Test Scenarios Generally Is a Bad Idea, IEEE Intelligent Vehicle Symposium (IV) 2020

Alessandro Calò; Paolo Arcaini; Shaukat Ali; Florian Hauer; Fuyuki Ishikawa: Simultaneously Searching and Solving Multiple Avoidable Collisions for Testing Autonomous Driving Systems, the Genetic and Evolutionary Computation Conference (GECCO) 2020

Florian Hauer; Ilias Gerostathopoulos; Tabea Schmidt; Alexander Pretschner: Clustering Traffic Scenarios Using Mental Models as Little as Possible, IEEE Intelligent Vehicle Symposium (IV) 2020

Alessandro Calò; Paolo Arcaini; Shaukat Ali; Florian Hauer; Fuyuki Ishikawa: Generating Avoidable Collision Scenarios for Testing Autonomous Driving Systems, IEEE International Conference on Software Testing, Verification and Validation (ICST), 2020

Florian Hauer; Tabea Schmidt; Bernd Holzmüller; Alexander Pretschner: Did We Test All Scenarios for Automated and Autonomous Driving Systems?, IEEE Intelligent Transportation Systems Conference (ITSC), 2019

Florian Hauer; Bernd Holzmüller: How Many Test Scenarios Do We Need for Testing Automated and Autonomous Driving Systems?, (German) TÜV Conference on Automated Driving, 2019

Florian Hauer; Alexander Pretschner; Bernd Holzmüller: Fitness Functions for Testing Automated and Autonomous Driving Systems, International Conference on Computer Safety, Reliability and Security (SafeComp), 2019

Florian Hauer; Bernd Holzmüller: Szenario-Optimierung für die Absicherung von automatisierten und autonomen Fahrsystemen, (German) FKFS AutoTest Conference, 2018