From the grave, Rembrandt inspires AI
In 1642, Rembrandt, or as his mom called him Rembrandt Harmenszoon van Rijn, was commissioned and finished the painting known as The Night Watch. The folks that commissioned the painting wanted to memorialize a military group. Each of the key members of the group portrayed in the painting paid 100 guilders each for a total of 1,600 guilders, which apparently was good pocket money back then. One key element of this painting is that it measures in at around 12 feet by 14 feet.
In 1715, the painting was moved to the Amsterdam Town hall. They had a special place in mind to mount the painting on a wall between two columns. Unfortunately, that key real estate was a bit too small to fit the huge painting. So, someone took scissors to it and trimmed it on three sides! The extra pieces were tossed aside. Luckily, Gerrit Lundens created a copy of the painting (no photography back then) in its full glory before the painting surgery took place.
A team at Amsterdam’s Rijksmuseum over the last few years thought this was a shame. They created a huge digital database of photos, scans, and x-rays of the painting. Key to this database were detailed photos of the Ludens copy to capture the missing pieces of the painting. Employing a neural network, they used this data to train a model on the colors and brush strokes used in the painting. This system also was employed to remove perspective mistakes in the copy versus the original painting. Using the photos of the missing pieces combined with the brush stroke data and the derived correct colors, the software was able to generate the missing pieces of the painting. The hardest part was generating the people removed from the left of the painting. The team now offers the “original” painting for all to enjoy via an online 4.8 gigapixel digital image.
I can see how the ability to analyze color and brush strokes could be a real asset for other art restorers around the world. I have seen enough art restoration shows to know, the key is the remove all the dirt, smoke and deteriorated varnish on a painting to reveal the original colors. Then, learning the real colors and the brush stroke techniques could help faithfully restore paintings. I also think the brush stroke technology might also be foundational technology for analyzing art work and historical documents for forgeries.
If researchers can apply AI to restore a painting, will we someday see the artist removed from the creation process all together? An art collective called Obvious accomplished just that. Using an AI technique based on a generative adversarial network and a data set of over 15k paintings, they generated a digital painting. From trial and error experiments, the team found that the best success was with portraits of people. The adversarial part of the network compares the differences between the generated painting and the human-created paintings in the database to “judge” the results. It selects the painting that matches closest to the human-created painting. Apparently the paintings in the database represented the old masters including Rembrandt. While it is not clear from my research if the AI-generated painting is actually rendered on canvas somehow, just a print of the digital image or just bits on a hard drive, it did sell at auction for $432, 500 (plus commission of course) in 2018. For that kind of cash, it should come with a very nice easel. In its own free house. I guess this price must be due to the novelty of the painting, because to my untrained eye, it looks pretty awful. But just recently, someone paid a lot of money for an invisible sculpture, so I cannot judge people’s taste in art.
Putting aside what our talented artists in the world think about AI-generated paintings, what would Rembrandt think? He died in 1669 at the ripe old age of 63, but as my grandma used to say, “He would probably be spinning in his grave.” I am not sure where that phrase came from or what it actually means, but I guess that he would be furious. On the other hand, I think he would be very happy that his original masterpiece has been restored (at least digitally).
