Understanding the role of AI and how to use data – the transcript

In my blog “Simulating the unexpected” I discussed how autonomous drive engineers employ our AI-driven solutions to find unexpected driving scenarios. I sat down and discussed how HEEDS is used in that industry and others. The podcast is here. For those that would rather read about this topic, here is the podcast transcript:

[00:00] Thomas Dewey: Welcome to AI Spectrum where we discuss an entire range of Artificial Intelligence topics. I’m Thomas Dewey, your host. In this series, we’re talking to experts across Siemens that apply AI technology to their products. Today, we’re very happy to have Roberto D’Ippolito, Product Manager at Siemens who works on Digital Factory Optimization solutions. Welcome, Roberto.

[00:23] Roberto D’Ippolito: Hello, Tom. Thank you for inviting me.

[00:26] Thomas Dewey: No problem. So, I thought I’d get started at a higher level before we dive into the work that you do at Siemens, and I wanted to start with what inspired your personal interest in AI?

[00:40] Roberto D’Ippolito: Well, I have been always a very curious person and especially I’ve been curious about how machines can mimic human behavior. More specifically, how they can learn how we do things from large amounts of information and possibly figure out somehow what is the relevant part that we are interested in too. So, I think I got enlightened when during the university studies, I stumbled into an exam talking about neural networks. At the time, it was maybe too early to do anything with them, but that’s where it all started.

[01:20] Thomas Dewey: Yes, that seems to be a common theme that people are either in college or somewhere and then all of a sudden they accidentally come across something that really piques their interest. How do you see the effects of AI on the industrial industry in general?

[01:36] Roberto D’Ippolito: Well, personally I believe that AI can serve a number of purposes if it is used properly. We can use a hammer to fix something or to crash it. AI is a little bit like this, it’s a tool. It’s a very powerful tool in our hands and it can have a large impact on how we do things if we use it properly. I believe that the largest impact will be that it will allow engineers to focus more and more on their engineering tasks without the need of becoming experts in scripting, in AI, or in too many other things that may drift them away from the engineering task they have. But I think if we stop thinking that AI is a magic wand that can do everything we want then we can definitely benefit from I would say it’s the real magic behind the scenes.

[02:36] Thomas Dewey: That’s a great observation. So, you’re the product manager of a product called HEEDS. Can you describe the product and how you use AI ML?

[02:45] Roberto D’Ippolito: Yeah, HEEDS is a product of the DISW portfolio. And it is a special type of tool, as it can be considered a little bit as the glue that automates the execution of simulation tools together, and provides the intelligence that exploits the simulation results to find better designs faster. So, large companies like Ford or Becton Dickinson say that HEEDS is a game-changing technology because it helps them to improve their development product, and also get quality much faster. So, in a nutshell, HEEDS is a powerful automation and design space exploration package. That’s the core DNA of the product and it allows the simultaneous execution of simulation tools. So, we are not doing simulation ourselves but we are using the simulation tools and automating them. And together with this, also the design space exploration. And that’s where our AI interest comes across. HEEDS is a tool really, that can be used by all designers, all types of engineers from beginners up to the very expert ones in all the phases of the product development process. So, it’s a rather transversal type of tool and it can really help design engineers to do more of their engineering job. And I will say since last year, finally, we have been extending the product with a number of AI techniques that I hope I can share more with you now.

[04:40] Thomas Dewey: It’d be interesting to hear what those techniques are.

[04:43] Roberto D’Ippolito: The techniques that we have are mostly focused on machine learning. You know that AI is a broad field that includes a lot of different things that go from expert rules, decision trees down to clustering, classification, and down the road also to deep neural networks. In HEEDS, we have been focusing mainly on machine learning and in the near future also on deep neural networks and the world around them. More specifically, I will say machine learning has always been in the HEEDS’ DNA from the very beginning. As a tool, HEEDS provides a cutting-edge optimization strategy called SHERPA, which is pretty well known. It’s one of the best optimization strategies on the market, and in its core parts, it is a machine learning strategy. It effectively learns automatically and adapts to the problem that you have. So, if you want to optimize a component, a mechanical component then you either know all the algorithms that are available outside there, and you know how to tune them or you can have SHERPA that does this for you. And in this sense, it is part of the machine learning approach that we want. So, this brings quite a big boost in the productivity to the engineer because they don’t need to be an optimization expert. Essentially SHERPA being a machine learning capability, learns about the problem and adapts to it. So, I will say this is a big benefit that you can get. But of course, if you are an expert, then you can get even more benefits from using it. I would say that the key benefit that we can get from AI in general, and machine learning in particular, is that it learns your problem for you but it’s not kicking you out of the process. Many people maybe fear that AI will take our jobs tomorrow, I’m not one of those. I think that AI machine learning will be a really powerful tool to assist us to do our job better. So, in this sense, machine learning automates a lot of the knowledge discovery process, so we can learn more, better, and make more sense out of it, than what we can do now by ourselves only.

[07:30] Thomas Dewey: That’s a good approach. Use AI to augment existing jobs, not replace jobs, basically. And so, when people employ HEEDS – you kind of mentioned a little bit about the benefits – but the AI or particularly or the ML portion of HEEDS, how does that help the customer get the product out and increase their productivity?

[07:54] Roberto D’Ippolito: Being SHERPA part of our DNA, we have decided to bring this concept to other tools that we have. So, we have developed a new machine learning strategy that when applied to engineering problems allows extracting information that is different from the one that SHERPA does. SHERPA is made for optimization, but not everyone wants to do optimization. Many people want to find their critical cases, others want to find where their solution is feasible, so it’s not violating constraints, or where discontinuities are. Think about, for example, autonomous vehicles. In the case of autonomous vehicles, a designer wants to know when the vehicle is failing. Can you imagine if you are in a car and you don’t have full control of the car because you are at full automation in autonomous vehicles? You want to make sure that the vehicle is behaving properly all the time in every condition. And to do this, you need to really test a lot. What if you can have a tool that automatically finds these cases? That will be helping a lot without you being necessarily an expert on all the specific conditions that will drive your autonomous vehicle in a critical state. So, what we have developed is a machine learning strategy that we call adaptive sampling, which is again, learning the problem from you. That is, it learns about the design and it continuously maximizes the amount of information and knowledge that is extracted from the simulation process. So, you have your autonomous vehicle, you put it in a scenario where you want it to behave properly, and you run a simulation out of it. You want to make sure that you learn the most out of this process. And once you have it, then you can learn where you need to improve your product if you are having an accident or something like this. So, there are, of course, a lot of applications where this can be of use. For example, digital twins, edge case detection, feasible regions, surrogate modeling, and so on. All these can get a very big benefit out of these approaches.

[10:33] Thomas Dewey: You gave the example of autonomous vehicles. Are there any other examples and industries that you can talk about that customers apply your technology?

[10:43] Roberto D’Ippolito: Well, autonomous driving is a big challenge so if you allow me, I would like to spend a few more words on that because it’s really putting a big challenge on top of the designers. The potential benefits that it can bring are known to everyone, but having an autonomous vehicle drive safely and comfortable is really a complex engineering challenge. And we can have engineering challenges that are complex in many other fields, in drones, in autonomous vehicles of any kind. It’s not only a car; it can be a boat, it can be a plane. And all these challenges are really putting the engineer in a difficult situation to find these unpredictable situations where you can have extreme weather, or a pedestrian jumping in front of the car from a blind spot. And when the interaction with the other vehicles on the road may turn out to become dangerous, then you need to know what to do. Finding and detecting all these cases is definitely not an easy job and it requires millions of simulations to be performed if you want to gain enough confidence, that you’re not committing a mistake. How can machine learning help these situations? Well, other cases that you can think of maybe in the design of complex systems like turbines for energy production, there can be any other complex system in a car or in electronics. Similar situations will always be tested against adverse conditions and whenever you need to find something that is a little bit a needle in the haystack to find, then machine learning can help you out. In this sense, the technique that we have developed in HEEDS allows you to really focus on the modeling aspect. So, what the engineer has to do is prepare a very accurate and good simulation model of what he needs to simulate and understand, and then once it is there, then you can automate the execution of this tool and give it to the machine learning algorithm. The machine learning will dig into this, will automate the execution of this simulation tool, and extract information out of it.

[13:15] Thomas Dewey: So, basically, in the normal simulation you’re throwing known test cases, but in the case of autonomous vehicles of any kind it sounds like more what you’re looking for are the outliers.

[13:27] Roberto D’Ippolito: We are looking for these situations that we call edge cases that are a little bit outliers, but they are also putting the vehicle into a challenging scenario. So, it may not be like an extreme weather condition, it can be just rainy, a bit wet, and then suddenly a pedestrian jumps on in front of the car from a place that you don’t see. So, this is maybe not necessarily an extreme situation, but it’s definitely an undesirable event, something that you want to avoid. How you find all these cases, all these situations that’s the challenge that we can delegate to the machine learning. So, the process is, I will say again, rather easy. All starts with the designer focusing on the modeling and simulation, and then once the model is ready, for example, let’s think about the lane change situation for an autonomous vehicle. This is a situation where you have an autonomous vehicle that is going on the road and then there is another vehicle that is about to change the lane and move in front of your vehicle. What happens in this situation? Either you brake, you change the lane or you just do nothing because the difference in speeds between the two vehicles would never give an impact. You need to test this type of condition at different speeds, different weather conditions, different lightning, and many other parameters that can characterize the scenario. However, the designer needs to find the challenging ones and answer questions like “If a car is moving in front of my car, at what speed is this dangerous? Can the braking system brake hard enough to avoid the collision or should I consider another maneuver to avoid the other vehicle?” These types of questions usually are answered by running millions of simulations, really hunting for the specific condition. But in a machine learning environment, adaptive sampling will allow you to learn from your problem, understand the physics, understand the context, see where the dangerous situations are, and hunt for these very particular cases. And then they will present you with a map of where these edge cases are. And that’s, I would say, the nice part of machine learning because you’re still in charge of what you need to do and what you need to learn, but machine learning – and in this case, adaptive sampling – is presenting you with a model of the knowledge and a model of the information which is where you have an accident or where you have a dangerous condition that is generated automatically. So, you don’t need to go and hunt for this, you can save a lot of simulations and you can get an accurate representation of where the problem is. Think about it a little bit like autofocus in a camera. Autofocus at the beginning, maybe the picture is blurry and you don’t see all the details, machine learning starts with this. Starts with the blurry picture of your problem and then automatically focuses in order to gain as much information as possible from the simulation tool. So, once you have this picture, this I’d say map of your knowledge, then it’s again, the time for the engineer to decide what to do. It’s fine, it’s a good situation, or should I change my design?

[17:23] Thomas Dewey: And I think people in the audience that aren’t familiar with simulation, when we’re talking millions or billions of simulations, once you reach that level there’s probably a good chance that you’ll never complete your product because it will take too long to simulate. So, the idea of narrowing down your simulation space and creating a lot fewer simulations is a very powerful thing in a production cycle.

[17:48] Roberto D’Ippolito: Yeah, of course, there is no free lunch here, don’t be misled by this. Machine learning is a powerful tool, but it still needs simulations, it still needs data to come in order to understand and to learn.

[18:03] Thomas Dewey: That was my next question, if the user simulations aren’t very adequate and is there a way to augment the data and where does the data come from for the training?

[18:15] Roberto D’Ippolito: In this case, data can come from different sources, it can come from experimental information like a recording of the car driving or experiments that you do on your turbines, you gain a lot of information. And these data can be used to discover potential cracks in your structure or potential glitches in your control algorithm that may lead to an accident and so on. But more in general, it can come also from simulations. So, having simulations that are as close as possible to the real world will produce a lot of data in a much cheaper way than the experimental part and will allow the designer to get the information that is needed to keep the passengers safe and comfortable. So, again, the more complex the problem becomes, the more simulations you will need, the more tests probably you will need, but overall, if we use machine learning, you will need a lot fewer simulations and a lot fewer tests, than without it. And that’s because machine learning maximizes the amount of knowledge and information that can be extracted from each simulation from each test that you perform.

[19:44] Thomas Dewey: Any customers in your work looking at synthetic data to augment the database?

[19:52] Roberto D’Ippolito: HEEDS is a tool that is made to be used by a lot of different customers. We do work with customers in the car manufacturing and in the autonomous vehicle context, of course, and they use experimental data like camera recordings, sensor recordings to build a digital twin of the car, of the vehicle, or of the situation. So, this is a way to support those customers by means of links to bring all the data together, provide to an algorithm that is AI-based and that can make sense out of this data, find the implicit relationships, and share it with the designer to make the right design decisions. So, yes, there is a lot of data that can be used, and probably it’s there without being leveraged yet, that machine learning can go and crunch to find all these relations that the designer needs to have a better design and faster.

[20:58] Thomas Dewey: That’s interesting. So, basically, you’re saying that some customers are sitting on some data they don’t even know that they can use in your solution?

[21:08] Roberto D’Ippolito: Yes. I will say yes. And it’s not necessarily their fault. Sometimes these powerful techniques are very hard to use. AI and machine learning, you can do a lot of good things, but I believe that the main challenge with these techniques to use the data that you’re sitting on is the packaging, how they are brought to the desk of the engineer. So, there is the main challenge in packaging a powerful set of technologies into a product that can be used by all engineers in all stages of the design process. That’s not easy, you don’t need to have only a powerful engine, you need to really make sure that is controllable by the person that needs to use it. So, my challenge as a product manager is to provide an easy to use, but yet powerful user interface of cutting-edge mathematics behind the scenes and deliver all these to both experts and non-experts. If I meet this challenge then this data can be really a kind of treasure to leverage for the engineer. It’s not easy, but especially when dealing with something quite innovative like AI and machine learning, I think that we really need to take the challenge. And I will say that so far, we have done quite good work in delivering for optimization purposes. And for now, for autonomous vehicles and for these machine learning techniques we are doing our best to make it easy so that you don’t need to be an expert to use machine learning, but you can make it immediately productive for you.

[23:03] Thomas Dewey: Speaking of challenges, were there any other challenges that you found trying to utilize AI and ML in your software product, other than the usability?

[23:16] Roberto D’Ippolito: I think also this myth that AI is a kind of magic wand that can do everything that you want. A lot of people believe that if it is AI-powered, then it can do some magic and give you the solution in a snap of a finger. And then maybe if you open the box, you see that it’s not really AI, but it’s a list of if/then/else clauses. And so, I think the challenge is also with the cultural mindset that needs to change a little bit to use AI as a tool and not as a solution and really may leverage the power that these tools can put in your hands. So, in a way, AI is kind of giving you some superpowers, but you need to make sure that you’re not relying too much on these techniques to be the only thing that would solve your problem. You still need to focus on the engineering problem, on the engineering challenge and leverage these techniques for the best for your problem.

[24:27] Thomas Dewey: That’s a real common theme with everybody involved with AI. I just attended a seminar and that was the clear message to focus on your business problem and let AI be the tool that helps you solve it but don’t focus on AI.

[24:43] Roberto D’Ippolito: Yeah. Definitely, it’s a mindset shift that has to happen to less believe it’s a magic thing. It is not magic, but it can be very powerful.

[24:52] Thomas Dewey: If you can tell people what your thoughts are on applying these AI ML technologies to your product going into the future.

[25:04] Roberto D’Ippolito: I think in the future these AI techniques will become better and better in supporting engineers in their daily tasks. So, engineers eventually will be able to do better things because smarter techniques and faster computational times will allow them to have more information, to make better decisions, to know and discover information that they can do also discover, but it will take too much time. The use of AI will make this information available in a much faster way. So, I believe that in the future AI technologies will be better and better at supporting engineers to do their work smarter and faster. But they will never replace the engineer himself or herself. AI cannot capture the creative process of the human. It can mimic it, but it cannot replace it. So, we still need engineers, very good engineers and with AI techniques in the future, they will become much faster in delivering high-quality, safe products.

[26:22] Thomas Dewey: I think that’s an excellent message and a good way to wrap up. So, thanks so much. Today I learned a lot and I’m sure our listeners will too. And thanks for joining us, Roberto.

[26:34] Roberto D’Ippolito: Thank you Tom for giving me the opportunity. I appreciate it.