Multi-pillar approach for safety validation of automated vehicles

The future of transportation promises to make life safer and more mobile for everyone, with positive economic results. However, to realize that promise, it is necessary to test new vehicles and every subsystem of their architecture, especially as the subsystems become smarter and highly complex.
The increased complexity necessitates a radical change of test methods and new concepts for comprehensive vehicle verification and validation in both the physical and the virtual world, which is captured in new regulations.

In this sense, in February 2021, the UNECE (United Nations Economic Commission for Europe) presented the New Assessment/Test Method for Automated Driving (NATM) – a framework, which introduces a multi-pillar approach for safety validation of automated driving (see Fig. 1) NATM, Guidelines.

Furthermore, in August 2022, the EU-Commission adopted the regulation 2022/1426 laying down rules for the application of Regulation (EU) 2019/2144 of the European Parliament and of the Council as regards uniform procedures and technical specifications for the type-approval of the automated driving system (ADS) of fully automated vehicles 2144, 1426.

What is the multi-pillar safety validation of AV?

The multi-pillar safety validation of automated vehicles specifies 5 certification pillars, which are supporting the safety argumentation. In addition to the three well-known pillars (track testing, real-world testing and audit), the regulation mentions virtual testing and in-service monitoring.

In this blog, the verification, validation, certification and assurance are defined/described as follows:

• Verification: is an activity that determines whether a system meets the requirements, answering the question: “Did we build the system right?”
• Validation: is assessing if the system meets the end user needs, answering the question: “Did we build the right system”. On the other hand, model validation is evaluating how well the model represents reality.
• Assurance: is justified confidence that the system functions as intended.
• Certification: determines whether a system conforms to a set of criteria or standards.

In case of autonomous vehicles, the scientific community realized early that only real-world testing – using mileage-based coverage – is not feasible, from an economical and technical point of view.

One of the main reasons is that during real world-tests, safety relevant events, happen very rarely. Therefore, it became obvious that virtual testing will play a key role in the certification of automated driving systems. The outcome of virtual testing can contribute to the safety argumentation if and only if the simulations are credible, this is briefly discussed in the next section and are detailed in NATM, Guidelines.

Virtual testing and simulation credibility assessment

In the case of virtual testing, we replace one or more physical elements with a simulation model.
Virtual tests used for automated driving systems (ADS) validation can achieve different objectives, depending on the overall validation strategy and the accuracy of the underlying simulation models.

Some of the objectives could be:

• Provide qualitative or statistical confidence in the safety of the full system.
• Provide qualitative or statistical confidence in the performance of specific subsystems/components.
• Discover challenging scenarios to test in the real world (e.g. real-world tests and track tests).

In contrast to all its potential benefits, a limitation of this approach is in its intrinsic limited fidelity of the models. As models can only provide a coarse representation of the reality, the suitability of a model to satisfactorily replace the real world for validating the safety of ADSs needs to be carefully assessed.

Therefore, the credibility of the simulation models and simulation environment shown in Fig. 2 shall be assessed to determine the transferability and reliability of the results compared to real-world performance.

In addition to simulation models and simulation environment, the credibility assessment is extended to the model and simulation management, too. All these aspects are captured in Fig. 3.

Read Siemens white paper

To learn more about the autonomous vehicle’s development workflows, read this white paper, that describes in more details:

• The challenges related to AV development
• the vehicle engineering workflow
• The scenario-based testing of autonomous vehicles and scenario-based testing workflows
• The simulation credibility assessment workflow

Ongoing activities and relevant regulations

Starting from October 2022, RDW (the Netherlands Vehicle Authority), JRC (European Commission’s Joint Research Center) and Siemens Industry Software Netherlands B.V., worked closely together to understand how to interpret and how to apply the new regulation especially, how to apply the credibility assessment framework.

For a better understanding, a specific use-case (automated valet parking) has been considered and each step of the simulation credibility assessment framework has been applied to this use-case.

The outcome of these investigations has been summarized in a credibility assessment handbook, which could be used as a guideline by automotive OEMs, interested to use virtual testing as a certification pillar.
Finally, for the interested reader a summary of the relevant standards and links are provided in the tables below.

UN-ECE regulation:

Regulation Number	Regulation Name	Purpose
1958 Agreement	ECE/TRANS/WP.29/2016/2	Agreement Concerning the Adoption of Harmonized Technical United Nations Regulations for Wheeled Vehicles, Equipment and Parts which can be Fitted and/or be Used on Wheeled Vehicles and the Conditions for Reciprocal Recognition of Approvals Granted on the Basis of these United Nations Regulations
R155	E/ECE/TRANS/505/Rev.3/Add.154	Uniform provisions concerning the approval of vehicles with regards to cyber security and cyber security management system
R156	ECE/TRANS/WP.29/2020/80	Uniform provisions concerning the approval of vehicles with regards to software update and software updates management system
R157	ECE/TRANS/WP.29/2020/81	Uniform provisions concerning the approval of vehicles with regard to Automated Lane Keeping Systems
NATM	ECE/TRANS/WP.29/2021/61	New Assessment/Test Method for Automated Driving (NATM) Guidelines for Validating Automated Driving System (ADS
Guidelines	ECE/TRANS/WP.29-187-10/2022	Guidelines and Recommendations concerning Safety Requirements for Automated Driving Systems

EU regulation:

Regulation Number	Regulation Name	Purpose
858	REGULATION (EU) 2018/858 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL	Type approval and market surveillance of motor vehicles and their trailers, and of systems, components and separate technical units intended for such vehicle
2144	REGULATION (EU) 2019/2144 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL	Type-approval requirements for motor vehicles and their trailers, and systems, components and separate technical units intended for such vehicles, as regards their general safety and the protection of vehicle occupants and vulnerable road users
168	REGULATION (EU) No 168/2013 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL	Approval and market surveillance of two- or three-wheel vehicles and quadricycles
1426	COMMISSION IMPLEMENTING REGULATION (EU) 2022/1426	Laying down rules for the application of Regulation (EU) 2019/2144 of the European Parliament and of the Council as regards uniform procedures and technical specifications for the type-approval of the automated driving system (ADS) of fully automated vehicles

Siemens involvement in EU projects

In addition to the ongoing research activities led by Siemens, RDW and JRC on multi-pillar approach for safety validation of automated vehicles, Siemens is involved in the following EU projects:

• FOCETA (FOundations for Continuous Engineering of Trustworthy Autonomy)
  Convergence of data-driven and model-based engineering. The underlying targeted scientific breakthrough of FOCETA lies in the convergence of model-driven and data-driven approaches. This convergence is further complicated by the need to apply verification and validation incrementally and avoid complete re-verification and re-validation efforts.
• AIthena (EU funded project number 101076754). AIthena is a research and innovation project on Connected and Cooperative Automated Mobility (CCAM) solutions that aims to build trustworthy, explainable, and accountable CCAM technologies.
• UrbanSmartPark (first project within “KIC Urban Mobility“, a European initiative supported by the EIT, the European Institute of Innovation & Technology). UrbanSmartPark project focusses on the development of automated on-street inner-city parking providing a broad range of possible parking-related services. Read this blog to find out more.
• SUNRISE (Horizon Research and Innovation Actions, Project No. 101069573 Call HORIZON-CL5-2021-D6-01). The SUNRISE project aims to establish a common safety assurance framework, interconnecting silos and making them collaborate in a harmonised way.
• DITM (Digital Infrastructure for Future-Proof Mobility, NextGenerationEU funded program). Aiming to support automated driving systems, the DITM partners are developing a system architecture for digital infrastructure, including the critical core technologies associated with localization, traffic services, digital maps, charging infrastructure, and continuous safety validation.
• ENVELOPE (EVALUATION AND VALIDATION OF CONNECTED MOBILITY IN REAL OPEN SYSTEMS BEYOND 5GS, is funded under the topic HORIZON-JU-SNS-2023-STREAM-D-01-01) will proceed to validate its CCAM Use Cases (UCs) through large-scale trials conducted across the three designated trial sites in Italy, The Netherlands, and Greece. In the ENVELOPE project these trials will be supported by European telecom operators, vendors, and vehicle providers/OEMs.

To learn more about the project, contact Alexandru Forrai (alexandru.forrai@siemens.com), Ph.D., fellow engineer and consultant at Siemens Digital Industries Software.