Compare CFD tools the right way (Part 1)

I have been involved in countless benchmarking situations, and I can tell you: it isn’t easy to compare CFD tools from different vendors. How do you pick the most accurate, fastest and budget-friendly one? Beware! There are a few things that are often done which lead to bad decisions. Let’s take a look at a few of those “how not to do it” cases:

Reading an online review by someone comparing multiple tools

Let’s be honest: how likely is it that someone comparing CFD tools from multiple vendors (from analyst level tools such as Simcenter STAR-CCM+ to designer level CAD embedded tools like Simcenter™ FLOEFD™) actually knows how to use all of them efficiently? Not to mention whether they test the same application and every tool has its strengths and weaknesses.

Since I love analogies, let me talk about a situation that we can all relate to:

I like cars and motorcycles. I have a driver’s license for both. However I don’t watch any Formula 1 or Superbike broadcasts. It’s just not something that would get me out of bed in the morning. I love their technology, but just watching cars going around in circles doesn’t excite me. However, I would love to go on a racetrack and have a go myself.

Someone could put me in a state-of-the-art Formula 1 car and briefly explain where to shift or break. I would do my lap in a record breaking worst time ever. Would you trust my judgment if I told you that the car was horrible and that I would have been faster in my own car? Is that the fault of the Formula 1 car?

No, of course not!

So why would you trust a report by a stranger? What is his background? What is his experience level with the tools? How rigorous was his test?  Did he get any training? Did he contact support if he had any issues?

Also what’s the role of the tester? Is it his job to do such tests and does he follow a standard set of tests? And most importantly, does he know what he is doing?

When there is no data to back the tester’s experience with each tool then there is no validity to the results of the test either.

Reading online forums about what users say about the tools

Let’s consider the reason why someone is posting in a forum about a CFD tool. That person is working maybe 8-10 hours a day, 5 days a week to get his tasks done. It is highly unlikely that they go online during work hours to chat about the tool he uses. Or during the weekend or his vacation. Typically you tend to write in forums when you have a problem and are looking for someone to help you. Sometimes while you are there you see a post and answer it.

That means, probably 90% of these posts are related to problems people are experiencing. It is unlikely someone goes online and waxes on lyrical about a product. This is usually only done for products you use for your hobby where you spend your time with during the weekends and your free time. Something you are generally loving and engaged in such as your camera, your nice car or motorcycle as well as other hobbies you might have, but not the tools you use at work like MS Outlook, MS Sharepoint or your CAD or CFD tool.

People are more likely to complain if something is not working than to post about such a tool on how great it is. I mean, when have you gone into a forum for a product you use at work and said “I just had a great day working with it.”? But you can remember the last time you had issues with it and went into a Forum and posted a question about something that is not working and were looking for help. This is natural to us humans, we take it for granted if something works and is not causing any issues but we complain if something is not working well and we need it to be fixed. Of course, otherwise we are stuck with our task and can’t get our work done.

So the only information you will find on these forums is basically based on a search for bad news. And guess what, if you search for car crashes, you will find car crashes. A more reliable source is to ask the vendor to provide a reference contact from another company. Then you can have a discussion – from one engineer to another. This would be even better if the contact is a colleague in a different country or division of your company.

Trust the vendors blindly (sorry colleagues)

Let’s be honest, sales people have their targets and need to sell their products. Understand that in the end, it is your company that has to make that decision. So compare CFD tools based on your criteria.

Therefore, define a typical set of tests that the tool would have to handle in your day to day work. Also, define a set of criteria that is important with weighting factors – not every company or department might have the same criteria. You probably know of the Weighted Decision Matrix and there are different ways of doing them.

In the end you will get a nice clean overview of the set of tools and their performance. You can still tweak the weighting if you think one criteria has become more important than another. The exercise should give you a clear idea at the end. In the end you might have to go with gut feeling for something you just liked more; for example, usability or the post-processing. But often this is only the case if it is a tight race between two of them.

Then test the software yourself. After all, it is you who has to use the tool. The vendors know how to use it and know all the tips and tricks to get the results. So it is important to see if you can handle the tool yourself. Is it as easy or fast as the vendor promised? After all, the vendor is the expert in the tool. For example, would you be able to handle the complexity of the tool? Or the way it is integrated into a CAD system if you don’t know how to use the CAD system?

Ultimately, it is you who will have to use the tool. Test it and make your own decision. This way you can compare CFD tools from different vendors in exactly the same way. Yes, it will take time and may cost something. But if you don’t you might end up with a tool that costs you more time and money than anticipated.

Watch out for the second part of the blog in a week from now… (to be continued)