The challenges with autonomous vehicle testing
One of the most critical aspects of autonomous car design, due to their complexity, is that you will never be able to physically test all possible accident scenarios. This is why simulation is so important in autonomous vehicle testing.
One company working to address safety through simulation is TASS International, which uses its expertise to develop safer and more intelligent vehicles. I recently spoke with three experts from TASS – Gwen Van Vugt, Frank Litjens and Paul Van Straten – about how the company is working to make autonomous vehicles safe.
In our conversation so far, we’ve discussed the vast amounts of testing required for autonomous vehicles. We’ve also discussed how simulation can allow for quicker verification of autonomous cars, as well testing the interface between humans and autonomous cars.
It’s the elusive “corner cases,” or rare cases that seldom occur, that can get you in trouble: for example, the sun shining directly into a car’s sensors or camera, inclement weather, or even a bicyclist crossing the street at night on a dimly lit street, as was the case in an unfortunate fatal Uber crash.
In this final part of our discussion, we move into further understanding comprehensive autonomous vehicle testing.
Edward Bernardon: Autonomous driving requires a massive amount of validation and verification cycles. And, as you go from Level 1 to 5 in a car, it goes from 100 scenarios to many millions. If you have to collect that data on the road, say at the rate that Tesla is currently collecting, it could take dozens of years. Therefore, there seems to be an obvious value in simulation technology.
Gwen Van Vugt: Yes, that’s where simulation comes in, because there’s no way you can test all these scenarios in the real world. It’s not only about collecting the data once, but adapting that collected data to your system and retesting it using the same data, and that is impossible in the real world.
You’ll never be able to redrive billions of kilometers to get to that same scenario again. You will never have all parameters under control, for instance, the weather today is cloudy, but tomorrow it’s sunny. So you have to bring things to the virtual world, and that’s what you’re seeing.
At this moment, there are many virtual simulation startups. It is a clear sign that there’s a big market for TASS, a market where we are a mature player with over 30 years of experience on the simulation side. We’re riding the wave of automated driving.
It’s not only about creating a virtual world. It’s also about sensor modeling and simulation, and to ensure it’s as accurate as possible in different physical domains like radar, LIDAR, ultrasonic, GPS, etc.
Another big topic where I think we have an advantage over startups is that you have to create databases of different realistic scenarios. That’s an area the TASS and Siemens combination, together with Mentor, can address.
Paul Van Straten: It’s not just important to collect regular driving data, but you also need to collect data on dangerous situations, and this can be costly. In testing and validation of any system, what you’re actually looking for is the “corner cases” as we call it.
Gwen Van Vugt: For example, if you envision a big room where most of the space is in the middle of the room – that’s not the interesting space, as you know that your system will already perform in that space. The interesting thing is how your system performs at the edges of the operating space, and that is where the corner cases come in.
But when you start driving, even as a human being, often you’re driving before you have experienced any critical situations that could possibly happen only a few times in your entire life – at least if you drive normally.
Human drivers are like Google and Tesla; they’re collecting massive amounts of data, but 98 percent of that data is of no interest, because it’s just normal driving situations.
You have to either collect an enormous amount of data, or the only way to test is to bring scenarios into simulation, start varying the scenarios in a virtual world and then search for your corner cases.
Edward Bernardon: How do you find the corner cases?
Gwen Van Vugt: Actually, it’s like the technology you use for optimization routines. You place seeds, which can come from accident databases. There are accident databases that exist in the world. One of them is the GIDAS 3D accident database, a German in-depth accident study. They have thousands of accidents mapped into a database, and we have an interface to that.
This database has a parametric description of an accident in time and space, and we download that into our virtual environment and replay that accident. But the nice thing is that when we replay it, we can change everything. We can change the speed of your vehicle or the speed of the other vehicles. We can change the weather conditions or algorithms used by a controller.
So then from one accident case, we can blow it up into thousands of cases by just varying different parameters in that situation, and that’s basically how you build up a scenario of databases.
Then by utilizing testing methodologies like Monte Carlo, there’s some standardized ways of searching for the weak spots of your systems response to a particular parameter. If there’s a lot of response, then it’s an interesting parameter. If there is no response, then there’s no need to search in that direction anymore. So there are standardized mathematical ways of searching for corner cases.
Bright future for autonomous vehicle technology
A study by ARK Investment Management indicates that if autonomous cars reduce driver errors consistent with reductions experienced by the airline industry when autopilots were introduced, motor vehicle accident rates will be reduced by 83 percent, from 1.1 to 0.2 deaths per 100 million vehicle miles traveled. As autonomous vehicles become safer than human-driven cars, one would expect that government regulation gradually moves toward making human driving illegal.
Getting there won’t be easy. Unlike airplane autopilots, escaping on land is not three dimensional; in the airspace, you don’t have to worry about a child or animal jumping in front of the vehicle. Response times are also much less on streets ― fractions of a second ― while you may have minutes to respond in the air. The number of scenarios that must be tested to ensure safety is also much higher for traveling on roads, as well as ensuring the safety of the land-based autopilot across an adequate number of scenarios.
Autonomous vehicle testing is certainly not an easy task.
In the future, governments won’t be able to certify car safety by requiring crash tests as they do today. Regulations will need to certify the simulations used to verify the performance of autonomous control systems at the heart of the future car. These regulations will not only help ensure our safety, but also unlock the full potential of the future car to deliver a new, exciting travel experience we’ll all look forward to and enjoy.
This concludes our interview series with experts from TASS International, a Siemens-based company, about autonomous vehicles. Special thanks to Gwen, Paul, and Frank for their time.
About the author
Edward Bernardon is vice president of strategic automotive initiatives for the Specialized Engineering Software business segment of Siemens PLM Software, a business unit of the Siemens Industry Automation Division. Bernardon joined the company when Siemens acquired Vistagy, Inc. in December, 2011. During his 17 year tenure with Vistagy, Bernardon assumed the roles of vice president of sales, and later business development for all specialized engineering software products. Prior to Vistagy, Bernardon directed the Automation and Design Technology Group at the Charles Stark Draper Laboratory, formerly the Massachusetts Institute of Technology (MIT) Instrumentation Laboratory, which developed new manufacturing processes, automated equipment and complementary design software tools. Bernardon received an engineering degree in mechanical engineering from Purdue University, and later received an M.S. from the Massachusetts Institute of Technology and an MBA from Butler University. He also holds numerous patents in the area of automated manufacturing systems, robotics and laser technologies.
