When off-the-shelf is not enough: fine-tuning for industrial part classification

The hard part of running a simulation is not always the physics. Sometimes it’s something far more far more basic: figuring out what the parts are. A CAD assembly of hundreds of components sits there, and before any mesh is generated, every single one needs a name. Is that a bushing or a bearing? A stud or a bolt? The answer comes either from metadata conventions that nobody keeps consistently, or from manual inspection that nobody volunteers for. Either way, days disappear before the real work begins.

In my previous blog, I explored how Vision-Language Models (VLMs) could step in and perform this classification automatically by looking at a CAD rendering and matching it to a text description. The results were promising as a proof of concept. But they also revealed a clear gap: off-the-shelf, zero-shot models struggle when parts look geometrically similar. A bearing and a bushing, for instance, can be nearly indistinguishable to a model that has never been trained on industrial components.

So the natural next question is: what if we teach the model?

When the model needs to go back to school: fine-tuning

Out-of-the-box AI models are trained on vast amounts of general data, which are images scraped from the internet, captions, photographs, diagrams. This gives them a broad, impressive understanding of the world. Ask one to identify a cat, a car, or a coffee cup, and it will rarely hesitate. But step into an industrial setting, and that generality starts to show its limits. A model that has learned from the internet has seen very few CAD renderings of mechanical components, and even fewer examples of the subtle geometric differences that distinguish a bushing from a bearing.

This is where fine-tuning comes in. Rather than training a new model from scratch, which would require enormous amounts of data, compute, and time; fine-tuning takes an already capable model and gives it a focused, targeted education on the specific domain you care about. Think of it like hiring an experienced engineer from a different industry: they already understand how to read technical drawings, how to reason about geometry, how to approach problems systematically. You don’t retrain them from zero. You simply give them a period of focused exposure to your specific parts, your specific context, your specific challenges, and they adapt.

Technically, fine-tuning works by continuing the training process on a new, smaller, domain-specific dataset. The model’s existing knowledge is preserved, but its internal parameters are gently adjusted, nudged rather than overwritten, so that it becomes more precise in the areas that matter to you. The result is a model that retains its general capabilities while developing a sharper, more reliable eye for the task at hand.

It is worth distinguishing fine-tuning from another popular approach: Retrieval-Augmented Generation (RAG). RAG does not change the model itself at all. Instead, it gives the model access to an external knowledge base at query time, essentially letting it look things up before answering. RAG is powerful when the knowledge you need is textual, structured, and can be retrieved by description. Fine-tuning, on the other hand, is the right tool when the challenge is perceptual, when the model needs to learn to see something differently, not just know more facts about it. For visual classification of industrial parts, where the distinction between categories lives in subtle geometric features, fine-tuning is the natural choice.

Teaching from experience: dataset

But before any AI model can learn, it needs examples. Lots of them. Think of it like training a new apprentice in a workshop. You wouldn’t hand them a textbook and expect them to identify every part on the shelf. You’d show them real components, from different angles, over and over, until the recognition becomes second nature.

That’s exactly what we did here. We used a large, publicly available library of mechanical parts, thousands of components spanning 27 categories, from everyday fasteners like pins, screws and bolts, all the way to more specialized parts like gears, rotors and fork joints. For each part, the model was shown multiple views, just like turning a component in your hands to get a full picture of what it is.

One thing this dataset reflects very honestly is the reality of industrial part libraries: common parts like pins and screws appear thousands of times, while rare ones like rivets or fork joints appear only a handful of times. Just as a seasoned engineer has seen far more bolts than bespoke joints in their career, the model learns from what it is shown most.

From a generalist to a specialist

The AI model we started with is already remarkably capable out-of-the-box. It can look at an image and match it to a text description with impressive accuracy across a huge range of everyday subjects. The problem is that it was trained on the internet, not in a machine shop.

So rather than building a new model from scratch, we did something much more practical: we took this already-capable model and gave it a focused apprenticeship in mechanical parts. We showed it thousands of CAD renderings, each paired with the name of the part it represents and let it recalibrate its understanding of what makes a bearing look like a bearing and not like a bushing.

The beauty of this approach is that the model doesn’t forget everything it already knows. It simply sharpens its eye for the specific visual details that matter in an engineering context. No need to build anything from scratch, no need for a massive research team, just a focused training process on the right data.

And you can actually watch that learning happen. The chart below shows how all three model variants improve over the course of training: starting out uncertain and making large errors, then steadily converging as they build a more confident understanding of the part categories. The steep early drop is the most telling part: the model picks up the most important distinctions very quickly and then continues to refine its understanding more gradually over time. By the end of training all three variants have settled at a similar level, which is a sign that the learning has genuinely taken hold, not just memorized a handful of examples.

The key move here is treating the loss curve not as a technical metric, but as a story of learning, as the model starts uncertain, learns fast and then consolidates. That framing keeps it accessible while still giving the graphic a meaningful role in the narrative.

Results

The outcome was clear: a model that has been trained on industrial parts is dramatically better at recognizing them than one that hasn’t. Categories where the out-of-the-box model was essentially guessing, for example bearings, bushings, motors, fittings became categories where the fine-tuned model performed with high confidence.

Predictably, the model performs best on the parts it has seen the most examples of. Common fasteners like pins, screws and bolts are recognized reliably and consistently. Rarer parts with only a handful of examples in the dataset remain more challenging. This is not a flaw in the approach; it’s a reflection of a simple truth: the more examples you show, the better the recognition becomes. And that is entirely within the control of the user.

A small number of categories like fork joints and clamps remain tricky even after training, because they look so similar to other parts and often underrepresented in the training dataset. But even here, the fine-tuned model is meaningfully better than where it started.

What this means for CAE workflows

Part classification is not an isolated task. It underpins automated defeaturing, boundary condition assignment, and assembly-level analysis. All of the steps that happen before a simulation even starts. Automating this step with a fine-tuned, open-source, locally deployable model means:

• No dependency on commercial APIs — the model runs in a controlled industrial environment
• Adaptable to your part library — fine-tune on your own components, not just generic benchmarks
• A building block for broader CAE automation — combine with the tools we explored in previous blogs to create end-to-end intelligent workflows

The fine-tuning technical work featured in this post was carried out by Cosmin Ivan. I am grateful for his technical expertise and effort that made this project possible.

Disclaimer

This is a research exploration by the Simcenter Technology Innovation team. Our mission: to explore new technologies, to seek out new applications for simulation, and boldly demonstrate the art of the possible where no one has gone before. Therefore, this blog represents only potential product innovations and does not constitute a commitment for delivery. Questions? Contact us at Simcenter_ti.sisw@siemens.com.