{"id":716,"date":"2018-11-28T05:57:21","date_gmt":"2018-11-28T13:57:21","guid":{"rendered":"https:\/\/blogs.plm.automation.siemens.com\/t5\/Digital-Transformations\/The-challenges-with-autonomous-vehicle-testing\/ba-p\/546025"},"modified":"2026-03-26T12:05:09","modified_gmt":"2026-03-26T16:05:09","slug":"the-challenges-with-autonomous-vehicle-testing","status":"publish","type":"post","link":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/the-challenges-with-autonomous-vehicle-testing\/","title":{"rendered":"The challenges with autonomous vehicle testing"},"content":{"rendered":"<p><P>One of the most critical aspects of autonomous car design, due to their complexity, is that you will never be able to physically test all possible accident scenarios. This is why simulation is so important in autonomous vehicle testing.<\/P><\/p>\n<p><P>One company working to address safety through simulation is TASS International, which uses its expertise to develop safer and more intelligent vehicles. I recently spoke with three experts from TASS \u2013 Gwen Van Vugt, Frank Litjens and Paul Van Straten \u2013 about how the company is working to make autonomous vehicles safe.<\/P><\/p>\n<p><P>In our conversation so far, we\u2019ve discussed the <a href=\"https:\/\/community.plm.automation.siemens.com\/t5\/Digital-Transformations\/What-will-it-take-to-make-autonomous-cars-safe\/ba-p\/507329\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">vast amounts of testing<\/A> required for autonomous vehicles. We\u2019ve also discussed how <a href=\"https:\/\/community.plm.automation.siemens.com\/t5\/Digital-Transformations\/Simulation-for-quicker-autonomous-car-safety-verification\/ba-p\/519142\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">simulation can allow for quicker verification<\/A> of autonomous cars, as well <a href=\"https:\/\/community.plm.automation.siemens.com\/t5\/Digital-Transformations\/Simulating-your-way-to-a-friendlier-autonomous-car\/ba-p\/528030\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">testing the interface<\/A> between humans and autonomous cars.<\/P><\/p>\n<p><P>It\u2019s the elusive \u201ccorner cases,\u201d or rare cases that seldom occur, that can get you in trouble: for example, the sun shining directly into a car\u2019s sensors or camera, inclement weather, or even a bicyclist crossing the street at night on a dimly lit street, as was the case in an unfortunate fatal Uber crash.<\/P><\/p>\n<p><P>In this final part of our discussion, we move into further understanding comprehensive autonomous vehicle testing.<\/P><\/p>\n<p><P><STRONG>Edward Bernardon: Autonomous driving requires a massive amount of validation and verification cycles. And, as you go from Level 1 to 5 in a car, it goes from 100 scenarios to many millions. If you have to collect that data on the road, say at the rate that Tesla is currently collecting, it could take dozens of years. Therefore, there seems to be an obvious value in simulation technology.<\/STRONG><\/P><\/p>\n<p><P><STRONG>Gwen Van Vugt:&nbsp;<\/STRONG>Yes, that\u2019s where simulation comes in, because there\u2019s no way you can test all these scenarios in the real world. It\u2019s not only about collecting the data once, but adapting that collected data to your system and retesting it using the same data, and that is impossible in the real world.<\/P><\/p>\n<p><P>You\u2019ll never be able to redrive billions of kilometers to get to that same scenario again. You will never have all parameters under control, for instance, the weather today is cloudy, but tomorrow it\u2019s sunny. So you have to bring things to the virtual world, and that\u2019s what you\u2019re seeing.<\/P><\/p>\n<p><P>At this moment, there are many virtual simulation startups. It is a clear sign that there\u2019s a big market for TASS, a market where we are a mature player with over 30 years of experience on the simulation side. We\u2019re riding the wave of automated driving.<\/P><\/p>\n<p><P>It\u2019s not only about creating a virtual world. It\u2019s also about sensor modeling and simulation, and to ensure it&#8217;s as accurate as possible in different physical domains like radar, LIDAR, ultrasonic, GPS, etc.<\/P><\/p>\n<p><P>Another big topic where I think we have an advantage over startups is that you have to create databases of different realistic scenarios. That\u2019s an area the TASS and Siemens combination, together with Mentor, can address.<\/P><\/p>\n<p><P><STRONG>Paul Van Straten:<\/STRONG> It\u2019s not just important to collect regular driving data, but you also need to collect data on dangerous situations, and this can be costly. In testing and validation of any system, what you\u2019re actually looking for is the \u201ccorner cases\u201d as we call it.<\/P><\/p>\n<p><P><STRONG>Gwen Van Vugt:<\/STRONG> For example, if you envision a big room where most of the space is in the middle of the room \u2013 that\u2019s not the interesting space, as you know that your system will already perform in that space.&nbsp;The interesting thing is how your system performs at the edges of the operating space, and that is where the corner cases come in.<\/P><\/p>\n<p><P>But when you start driving, even as a human being, often you\u2019re driving before you have experienced any critical situations that could possibly happen only a few times in your entire life \u2013 at least if you drive normally.<\/P><\/p>\n<p><P>Human drivers are like Google and Tesla; they\u2019re collecting massive amounts of data, but 98 percent of that data is of no interest, because it\u2019s just normal driving situations.<\/P><\/p>\n<p><P>You have to either collect an enormous amount of data, or the only way to test is to bring scenarios into simulation, start varying the scenarios in a virtual world and then search for your corner cases.<\/P><\/p>\n<p><P><span class=\"lia-inline-image-display-wrapper lia-image-align-center\" style=\"width: 400px;\"><img decoding=\"async\" src=\"http:\/\/blogs.sw.siemens.com\/wp-content\/uploads\/sites\/19\/2019\/09\/Autonomous-vehicle-testing-1.jpg\" alt=\"Autonomous vehicle testing.jpg\" title=\"Autonomous vehicle testing.jpg\" \/><\/span><\/P><\/p>\n<p><P><STRONG>Edward Bernardon: How do you find the corner cases?<\/STRONG><\/P><\/p>\n<p><P><STRONG>Gwen Van Vugt:&nbsp;<\/STRONG>Actually, it\u2019s like the technology you use for optimization routines. You place seeds, which can come from accident databases. There are accident databases that exist in the world. One of them is the <a href=\"https:\/\/www.gidas.org\/en\/willkommen\/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">GIDAS 3D accident database<\/A>, a German in-depth accident study. They have thousands of accidents mapped into a database, and we have an interface to that.<\/P><\/p>\n<p><P>This database has a parametric description of an accident in time and space, and we download that into our virtual environment and replay that accident. But the nice thing is that when we replay it, we can change everything. We can change the speed of your vehicle or the speed of the other vehicles. We can change the weather conditions or algorithms used by a controller.<\/P><\/p>\n<p><P>So then from one accident case, we can blow it up into thousands of cases by just varying different parameters in that situation, and that\u2019s basically how you build up a scenario of databases.<\/P><\/p>\n<p><P>Then by utilizing testing methodologies like <a href=\"https:\/\/www.sciencedirect.com\/topics\/neuroscience\/monte-carlo-method\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">Monte Carlo<\/A>, there\u2019s some standardized ways of searching for the weak spots of your systems response to a particular parameter. If there\u2019s a lot of response, then it\u2019s an interesting parameter. If there is no response, then there\u2019s no need to search in that direction anymore. So there are standardized mathematical ways of searching for corner cases.<\/P><\/p>\n<p><H3>Bright future for autonomous vehicle technology<\/H3><br \/>\n<P>A study by ARK Investment Management indicates that if autonomous cars reduce driver errors consistent with reductions experienced by the airline industry when autopilots were introduced, motor vehicle accident rates <a href=\"https:\/\/ark-invest.com\/research\/autonomous-vehicle-safety\" target=\"_blank\" rel=\"nofollow noopener noreferrer\">will be reduced by 83 percent<\/A>, from 1.1 to 0.2 deaths per 100 million vehicle miles traveled. As autonomous vehicles become safer than human-driven cars, one would expect that government regulation gradually moves toward making human driving illegal.<\/P><\/p>\n<p><P>Getting there won\u2019t be easy. Unlike airplane autopilots, escaping on land is not three dimensional; in the airspace, you don\u2019t have to worry about a child or animal jumping in front of the vehicle. Response times are also much less on streets \u2015 fractions of a second \u2015 while you may have minutes to respond in the air. The number of scenarios that must be tested to ensure safety is also much higher for traveling on roads, as well as ensuring the safety of the land-based autopilot across an adequate number of scenarios.<\/P><\/p>\n<p><P>Autonomous vehicle testing is certainly not an easy task.<\/P><\/p>\n<p><P>In the future, governments won\u2019t be able to certify car safety by requiring crash tests as they do today. Regulations will need to certify the simulations used to verify the performance of autonomous control systems at the heart of the future car. These regulations will not only help ensure our safety, but also unlock the full potential of the future car to deliver a new, exciting travel experience we\u2019ll all look forward to and enjoy.<\/P><\/p>\n<p><P><EM>This concludes our interview series with experts from TASS International, a Siemens-based company, about autonomous vehicles. Special thanks to Gwen, Paul, and Frank for their time.&nbsp;<\/EM><\/P><\/p>\n<p><P><STRONG>About the author<\/STRONG><BR \/><STRONG>Edward Bernardon<\/STRONG> <EM>is vice president of strategic automotive initiatives for the Specialized Engineering Software business segment of Siemens PLM Software, a business unit of the Siemens Industry Automation Division. Bernardon joined the company when Siemens acquired Vistagy, Inc. in December, 2011. During his 17 year tenure with Vistagy, Bernardon assumed the roles of vice president of sales, and later business development for all specialized engineering software products. Prior to Vistagy, Bernardon directed the Automation and Design Technology Group at the Charles Stark Draper Laboratory, formerly the Massachusetts Institute of Technology (MIT) Instrumentation Laboratory, which developed new manufacturing processes, automated equipment and complementary design software tools. Bernardon received an engineering degree in mechanical engineering from Purdue University, and later received an M.S. from the Massachusetts Institute of Technology and an MBA from Butler University. He also holds numerous patents in the area of automated manufacturing systems, robotics and laser technologies.<\/EM><\/P><\/p>\n","protected":false},"excerpt":{"rendered":"<p>One of the most critical aspects of autonomous car design, due to their complexity, is that you will never be able to physically test all possible accident scenarios. This is why simulation is so imp&#8230;<\/p>\n","protected":false},"author":45480,"featured_media":720,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spanish_translation":"","french_translation":"","german_translation":"","italian_translation":"","polish_translation":"","japanese_translation":"","chinese_translation":"","footnotes":""},"categories":[1],"tags":[21,25],"industry":[120],"product":[],"coauthors":[],"class_list":["post-716","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news","tag-autonomous-vehicles","tag-cars-of-the-future","industry-automotive-transportation"],"featured_image_url":"https:\/\/blogs.sw.siemens.com\/wp-content\/uploads\/sites\/19\/2019\/09\/Autonomous-vehicle-testing-1.jpg","_links":{"self":[{"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/posts\/716","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/users\/45480"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/comments?post=716"}],"version-history":[{"count":2,"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/posts\/716\/revisions"}],"predecessor-version":[{"id":721,"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/posts\/716\/revisions\/721"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/media\/720"}],"wp:attachment":[{"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/media?parent=716"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/categories?post=716"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/tags?post=716"},{"taxonomy":"industry","embeddable":true,"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/industry?post=716"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/product?post=716"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/blogs.sw.siemens.com\/thought-leadership\/wp-json\/wp\/v2\/coauthors?post=716"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}