At MoMoLab, we’re always on the search for new approaches to storytelling and interactive experiences. Our regular visits to museums and information centres often inspire us to explore different directions. Other times, these visits lead us to reconsider the how visitors are currently expected to engage with the visual and written content on display. For the past year, this one idea has been bouncing in the back of our minds; What if you could have a conversation directly with one of your favourite pieces of art? To test and evaluate the merits of this idea, we created an interactive experience prototype. This concept would enable visitors to learn about the history and importance of a specific artwork, straight from the subject featured in this painting.
Meet the Merry Fiddler
This painting seemed like an obvious subject for this prototype for a simple reason. When I first moved to the Netherlands, this painting was hanging in the hallway outside of my apartment. I always liked the warm expression on the musician’s face as he welcomed me home each day.
teaming up with Wintor
First, we needed a proof of concept. We collaborated with the team at Wintor, developers of an augmented reality tour app, to experiment with their software and see how far we could take this concept. We set up a prototype on location in a former metalworking factory in Hilversum, kindly offered to us by Fabrique Ludique.
the process
The video below documents the process we took in making the prototype.
In the weeks leading up to the on-site tests, we prepared the assets. This started with downloading a digital copy of Gerard van Honthorst’s 1623 painting from the Rijksmuseum’s online public catalogue. Using Photoshop, we segmented the painting into layers, taking advantage of the amazing new generative fill options offered in the more recent Photoshop updates. We researched the painting’s history, studied why van Honthorst chose his subject, and learned the significance of the artwork in the context of genre painting.
This became the basis for a short script, allowing the painting’s subject to speak for himself. We animated the different body parts of the Merry Fiddler using text-to-speech tools and face tracking software. We even imbued him with a slightly Italian accent in honour of the setting of the painting. This was possible thanks to Elevenlab’s latest multilanguage options.
In After Effects, we combined all the elements—voice, face tracking video, and animated layers of the body—creating a 3D scene extending beyond the frame of the painting. The scene we made became a representation a 17th-century Italian piazza and an atmospheric setting to learn about this painting.
We uploaded this scene to YouTube as a 360° video that you can watch below.
Moreover, we designed an interactive VR experience. This virtual environment enabling users to explore the environment with 6 degrees of freedom (6DOF), incorporating 3D UI elements and spatial audio to enhance the storytelling.
evaluating this interactive experience
Importantly, I should acknowledge that this was just a prototype. It was made on a tight research budget, using free versions of AI software, resulting in the somewhat clunky AI voices offered by these free trials. If we had a serious project and a party that wished to invest in this concept, I am fully confident we could make an exceptional exhibit or interactive VR experience. The resulting experience could include an AR tour with talking paintings running on visitor devices (of course powered with Wintor AR). Or if there was a bigger budget or the space for an VR exhibit, the resulting experience could be a stand-alone interactive VR exhibition that could be placed in a museum parallel to the physical versions of the paintings.
Each different implementation has its specific strengths and best fits specific conditions. Despite the differences, these experiences share something essential, they do not try to replace the beauty or value of the original artworks—they celebrate them by combining the warmth of human voices, the magic of high-tier animation, and the atmospheric settings created by sound design and 3D modelling.
how museums could benefit
For museums interested in developing a virtual collection that parallels their physical one, these concepts introduce entirely new ways for visitors to experience artwork and engage with art history. This is actually ideal. The best results would come from an experience curated by museums using scripts written by art historians, voiced by talented voice actors, and tied together with beautiful sound design and 3D modelling. This would be an modern interpretation that remains true to the original works . The digital museum exhibits could be site specific or become VR experiences downloaded to run people’s own VR/XR devices, no matter the location.
next steps for this interactive experience
Looking ahead, we have future goals for this project. We aim to develop a character AI model based on The Merry Fiddler, allowing users to engage in real-time unscripted conversations back and forth with their voices, like guided art tour chatbot. Also, we know that this next step in this process opens a new world of challenges, both technical and ethical. So for now, the scripted prompts of question and answer that we experimented with works good enough. Also, importantly, the scripted nature of the questions and answers gives museums control over the narratives presented by these characters, guaranteeing information that is factual and researched. Not manufactured by the black box of AI and unsubstantiated.
We are hoping that museums and visitor centres will recognize that this new way of storytelling can resonate with the next generation of museum-goers, presenting a modern way for conveying information to an audience. Bring art to life using new technical solutions can help keep these artworks relevant and enjoyable to all visitors.
If you or your organization would like to develop an interactive experience like the one created for the Merry Fiddler, be sure to reach out to us here.