End-to-end virtual verification and validation of vehicle automation systems

Introduction
Road vehicle automation functions, both in the field of advanced driver assistance systems (ADAS) as well as automated vehicles (AVs), require extensive testing for verification of the requirements and validation of the resulting functionality. In particular ‘safety of the intended functionality’ (SOTIF) needs assessment of the vehicle’s behavior in a large number of traffic scenarios to ultimately obtain type approval. To reduce testing costs and time, Siemens develops a toolchain for virtual verification and validation adopting a scenario-based approach, aiming to significantly decrease the number of real-world tests. This toolchain involves, among others, requirements definition, data collection, scenario detection and classification, critical scenario creation, and simulation[1] .
This blog first presents data collection, using a Siemens measurement vehicle. Next, the scenario extraction step in the toolchain is illustrated, enabled by Simcenter Autonomy Data Analysis. The blog concludes with some remarks on the generation of synthetic safety-critical scenarios, being the next step in the toolchain.
Data collection
Our data collection vehicles are off-the-shelf vehicles which are instrumented in-house with sensors, a data-acquisition platform, and a robust power supply system which ensures reliable data collection. The data to be collected concerns information about other road users (position, speed, and heading relative to the measurement vehicle) and information of the measurement vehicle (position, speed, and heading in a world-fixed coordinate frame).
One of the data collection vehicles is a Kia EV6, as shown in Figure 1. This vehicle is instrumented with a 360o Ouster OS1-64 lidar, four Leopard cameras which together provide a surround-view of the vehicle, and a Septentrio GNSS system with RTK-correction for accurate vehicle pose information. To ensure consistent timestamps for data being received from the different sensors and for necessary synchronization between the sensors, a GMR Masterclock device is used which operates based on the Precision Time Protocol (PTP). The data is recorded in a Simatic Industrial PC configured for data acquisition using ROS2, and is then further serialized to MCAP format.

Scenario extraction
When recording a large amount of driving data in real traffic, a large part of the collected data is actually uneventful. It is therefore important to evaluate if the data used for testing is challenging and diverse. Addressing this challenge requires two solutions.
First, we must be able to quickly identify relevant parts of the data.
Second, to provide evidence for a safety argument for the ADAS/AV and to understand if more data collection is necessary, we must be able to quantify how sufficient the amount of data is with respect to testing requirements. The cloud-based Simcenter Autonomy Data Analysis platform provides both these solutions through an automated analysis of the data, extracting key scenarios from the raw data logs, as illustrated in Figure 2.
Figure 2: Users can query the processed data for safety-relevant events
The cherry on top is the export functionality of Simcenter Autonomy Data Analysis: scenarios can be exported as an ASAM OpenScenario file for the traffic participant motion and a corresponding OpenDrive file for the road infrastructure, see Figure 3.
Figure 3: Several predefined scenario categories are automatically extracted from the driving data and are exported into OpenScenario and Opendrive for virtual testing
This export functionality allows for virtual testing of the ADAS/AV system in a simulation environment that is capable of ingesting the OpenScenario/OpenDrive files, such as our own Simcenter Prescan. This is illustrated in Figure 4, which shows a scenario concerning the ego vehicle that approaches a stationary vehicle at a traffic light and several cut-inn/cut-out scenarios. These scenarios were part of the footage captured by our measurement vehicle. The footage was analyzed in Simcenter Autonomy Data Analysis to extract the scenarios. These scenarios, as part of the extracted set of scenarios, were then exported as OpenScenario and OpenDrive files and subsequently imported into Simcenter Prescan. As a next example, Figure 4 shows several real-life scenarios imported in Simcenter Prescan.
Figure 4: Approach of a stationary vehicle and cut-ins (from right and left), extracted from real-world data, imported in Simcenter Prescan (traffic lights not included in simulation)
Scenario generation
Besides replay of the scenarios in Simcenter Prescan, the processed real-world data and extracted scenarios are used as a basis for massive virtual verification and validation to address the SOTIF safety standard (ISO 21448). This includes testing system weaknesses such as diverse illumination and weather conditions, exploring the parameter space within the extracted logical scenarios for known-unsafe scenarios, and analyzing road user behavior for creation of unknown-unsafe scenarios in the intended operational design domain (ODD) of the ADAS/AV. To learn more about our patented unknown-unsafe scenario identification methodology, see our dedicated blog post on this topic[2] and our technical papers[3], [4].
References
- Leveraging the Siemens toolchain: Testing and virtually validating advanced driver assistance systems and autonomous vehicle functions ↩︎
- Establish confidence in autonomous vehicle systems with patented Critical Scenario Creation framework ↩︎
- A Systematic Approach for Creation of SOTIF’s Unknown Unsafe Scenarios: An Optimization based Method ↩︎
- Unknown-Unsafe Scenario Generation for Verification and Validation of Automated Vehicles | IEEE Conference Publication ↩︎