The incredible pace of digital transformation

By Jenn Schlegel

Rembrandt, AI and the missing pieces of The Night Watch.

Rembrandt and AI. Now, I know, you are thinking: what in the world would Rembrandt have to do with AI? Today, in honor of Rembrandt’s birthday (July 15, 1606 in Leiden, The Netherlands), we are going to talk about just that.

Rembrandt and AI

Last September, I headed up to Amsterdam for one of my favorite tech events. (It was my first conference with “real live people” in almost 2 years.) One of the speakers was Robert Erdmann, Senior Scientist at the Rijksmuseum. He spoke about how he and his team “trained” AI neural networks to recognize certain pigment colors. They were taught to sort through millions of images of art from the collection, and even “paint” like Rembrandt. This work with the AI neural networks would be a significant factor in recreating the missing pieces of Rembrandt’s masterpiece, The Night Watch.

The missing pieces of The Night Watch

For those of you who don’t know the story of The Night Watch, it’s a good one. During the Dutch Golden Age, Rembrandt was one of the most popular (some say rebellious) painters of his time. In 1639, he received an important commission for a group portrait from Captain Banning Cocq and 17 other members of their civic guard — a local militia that defended Amsterdam mostly from the Spanish and kept all-around peace on the streets. Rembrandt was paid 1600 Dutch guilders which was a serious sum of money at the time. (This would be about 95,000 euro today. Quite the bargain in retrospect). Rembrandt finished The Night Watch in 1642. For the next 73 years, it happily hung in the banquet hall of the Kloveniersdoelen, the group’s guild house.

The Night Watch by Rembrandt
The Night Watch is staged like a performance rather than a face-forward portrait line-up. This was a major break from the convention of the time.

1715: The Night Watch doesn’t fit

In 1715, The Night Watch moved to the Amsterdam city hall. But…whoops, the giant canvas was too big to fit between two doors on the designated wall. The folks in charge decided to lop off a good 60cm chunk on the left side, some 7cm on the right, 22cm from the top, and 12cm from the bottom. Voilà, it fit just nicely. Unfortunately, it seems that nobody bothered to save the missing pieces. They completely disappeared.

Operation Night Watch

In the summer of 2019, the Rijksmuseum started Operation Night Watch, a grand-scale research and restoration project on the masterpiece. Centered around AI neural networks and ultra-high-resolution imaging, Operation Night Watch set a new standard in the world of art conservation techniques.

Operation Night Watch at the Rijksmuseum in Amsterdam
Most of Operation Night Watch took place in the public eye with experts working in a transparent glass chamber in the Rijksmuseum.

Enter big tech

To start, the team at the Rijksmuseum created a 44.8 gigapixel ‘base’ image. (You might think of it as a digital twin technology demonstrator, like our very popular Simcenter SimRod.) 528 different still photographs were stitched together digitally with the aid of AI neural networks. The AI networks recognized miniscule paint pixels and matched them precisely. From this, the team put the neural networks to work further on those missing pieces.

As Robert Erdmann explains in this movie, the team worked with digital copies of The Night Watch and a copy of the original painting by one of Rembrandt’s fellow painters, Gerrit Lundens. The AI neural networks were taught to recognize and correct perspective differences in the two paintings. Also, they could recognize the color scheme and project that onto the pixels of the missing pieces. As a next step, as Erdmann puts it, the neural network was “sent to art school” to learn how “to paint” like Rembrandt. By replicating the brushstrokes of the master, the team could make the best-possible digital twin of the missing pieces. And print the missing pieces for the general public to see.

The Night Watch by Rembrandt with the digitally printed missing pieces was briefly displayed at the Rijksmuseum.
The Night Watch by Rembrandt with the digitally printed missing pieces was briefly displayed at the Rijksmuseum.

An even better digital twin?

Although, the original painting with the digitally-printed missing pieces was only displayed briefly at the Rijksmuseum, it does show just how quickly digital transformation has taken on all aspects of daily life. Just 18 months later, the team undertook a crazy mission of capturing and stitching together an even-more detailed image. They used the 44.8 gigapixel base image as a guide to produce another even more detailed digital twin of The Night Watch. This time an ultra-high-resolution 717 gigapixel image – the largest and most detailed photograph of any artwork to date.

Go ahead: dream bigger

And you can experience it online. This image is four times sharper than its predecessor (the 44.8 gigapixel base image) and provided enough data so that researchers, conservation experts, and art historians could zoom into pinhead-shaped particles of Rembrandt’s pigments.

Details from the digital twin of The Night Watch
Minute cracks, individual paint particles, past retouch work, and brushstroke details of Rembrandt’s painting technique are as clear as day.

The more data, the better

Thanks to the rapid-pace of digital transformation and smart AI-driven technology, the Operation Night Watch team (and those AI neural networks) now have even more data to work with. With all this information at the digital-ready, the restoration team at the Rijksmuseum can make the best-informed decisions regarding future conservation plans.

As we move further into the new world of digital ecosystems, technology advancements like Operation Night Watch will show us the positive power of digital transformation for humanity as a whole.

Now, if they could only teach those AI neural networks to find those missing pieces from 1715…


And speaking of AI, here are just a few of the latest Simcenter AI resources:

Images courtesy of The Rijksmuseum.

Unleash innovation with AI/ML and Simcenter
This article first appeared on the Siemens Digital Industries Software blog at