Pau Waelder
Musician and digital artist Franz Rosati recently presented on Niio a series of three videos titled DATALAKE: GROUNDTRUTH (2024) in which he worked with AI models to generate mesmerizingly fluid landscapes that evoke chaos and disaster, but also regeneration and impermanence. Widely recognized on the international digital art and electronic music scene, Rosati creates a unique combination of visual landscapes, soundscapes, and cinematic narratives that embody the ideal of the total work of art (Gesamtkunstwerk) for a fully immersed audience.
In a previous interview for Niio Editorial, on the occasion of the launch of Rosati’s series LATENTSCAPE (2021), the artist offered us a general approach to his work. In this article, we focus on his most recent series, diving into his use of machine learning programs, his reflections about our current visual culture, and the elusive concept of truth.
Franz Rosati. DATALAKE:GROUNDTRUTH N1, 2024
This series brings to mind our fascination with disasters, which has been enhanced by digital technologies allowing the creation of ever more spectacular images of cataclysms in films and TV shows, and the growing tendency in mainstream media to feed us with disturbing images in the midst of the raging attention economy. Would it be correct to interpret the artworks as a commentary on our current media landscape?
I don’t know if I want to see the work as a commentary on the media landscape, this is not the very first target of the project.
For sure the theme of disaster, the cataclysm, and the overwhelming flux of visual information, has extended not only to the natural and ecological context, but also to the perception of a cognitive vortex. So the disaster as something disruptive, unexpected and out of control, is exactly what emerged during the production and the selection of the final materials.
I do not start any of my projects with a specific political or critical theme, it’s a pretty more subconscious process, I just start doing things and stop when something resonates with some parts of me, then I try to figure out why it resonates with me so much. This is why I’m very interested in understanding how my work might adhere to a vision of reality.
I think every artwork is political in some ways but I also think that every artwork must speak for itself, it’s not about releasing a statement through images and sounds, but instead leaving it there and observing how it resonates with the surrounding environment.
In this case, after months of generating materials and testing sounds, I decided that I wanted to catch the vibe of a visual multiverse inflated by images, that through an inescapable hyper-aesthetization process, disastrously decorate and beautify its own impermanent deflagration and collapse.
“After months of generating materials and testing sounds, I decided that I wanted to catch the vibe of a visual multiverse inflated by images that disastrously decorate and beautify its own impermanent deflagration and collapse.”
The use of split screens in some scenes reinforces the connection with newsreels and mass media. What do they bring to this project, in terms of visual composition and subtext?
The splitscreen aesthetics idea came out in an advanced phase of the work and was absolutely the most difficult part to do from a technical aspect.
I didn’t want to do it by using masks to tell AI where to put different things, which is a pretty common and straightforward approach. Instead I wanted to let AI interpret several words in the prompt, going back and forth to see how a comma or the weight of a specific word can alter the results and recall some features in the model. This was a pretty long process.
The screen is a very iconic and persistent element of our visual culture since Nam June Paik (and before him Guy Debord, Isidore Isou or Abel Gance) which was probably the first in visual arts to understand how powerful could it be, but today we have incredible artists such as Baron Lanteigne who is able deal with a simulacra of the screen in extremely powerful, subtle and unexpected ways, as in the content so in the setup of his works.
The splitscreen in this series is my “tribute” to the device, in its embodiment as a magic portal, a simulacra, a magic surface opening up to other worlds. A multidimensional device shaping our reality more and more, first in our homes, then covering walls and buildings of our cities, then fitting in our hands and pockets to slowly landing in the art galleries and exhibitions.
In my case I wanted multiple screens representations to be a strong part of the artwork: The television layout which is the one telling what’s happening in our world shaping our perception of reality as well as the LED wall or the stage design which is instead the most diffused example of mass worship during concerts and large events.
“The splitscreen in this series is my «tribute» to the device, in its embodiment as a magic portal, a simulacra, a magic surface opening up to other worlds.”
I want these elements to be almost like a living organism that is symbiotic with the natural elements, absorbs nature and is absorbed by nature. Virtuality and biology melted together.
So it is not just a way to visualize different contents and context. It is more about the splitscreen, the “news layout”, the LED wall, as a part of a new and complex hybrid ecosystem.
An interesting detail in these scenes of chaos and destruction is the presence of cars. Artist Iñigo Bilbao once stated: “Confronted with images of a disaster, in order for them to cause us a real (and perhaps pleasant) impact, we need to identify something among the chaos, a reference that informs us of the scale of the tragedy […] Apart from the most macabre evidence, cars offer an easily recognizable shape, allowing us to glimpse the seriousness of the accident, the power of the bomb or the height of the floods.” Is the presence of cars in these artworks connected to this sense of scale mentioned by Bilbao? Or maybe it is something embedded in the imagery of disasters that AI models replicate?
Cars and humans have a very accidental yet central role in this work, retrospectively. That wasn’t planned at all, my idea at the beginning was only focused on depicting this process of nature, architecture and technology struggling to coexist and find their place, fighting, morphing and twisting.
I tried everything to totally get rid of both humans, cars and vehicles, and keep only images of technology. For some reasons the CLIP (Contrastive Language-Image Pretraining) part of the code responsible for learning and connecting visual concepts from natural language supervision, was strongly trained with tons of images labeled as cars, vehicles and people or something strictly related. So the semantic domain of technology is not separable from cars and vehicles.
The observation made by Iñigo Bilbao is revealed to be dramatically true and fitting in this case. In the end I decided to keep cars and vehicles and play with them, giving them a role, as a reminder of the many biases we can discover in AI, and how that speaks so much about our world and how some peculiar aspects of our society will always percolate into the tools we use and back in a recursive process, shaping their functions and role in our evolution or progress.
This could lead to a very simple deduction: The tool is the message or at least the part of the algorithm generates part of the message.
“I wanted to let AI interpret several words in the prompt, going back and forth to see how a comma or the weight of a specific word can alter the results. This was a pretty long process.”
The soundtrack is a crucial element of the artworks, as it brings in a strong emotional element to the images. As a musician, how would you describe the role of sound in this series?
Sound is very central for me, for many years I’ve worked with materials gathered from streaming platforms, and social networks as well as field recordings or unusual sounds, trying to build some kind of accidental sound design and music from the social bubble as Burial first taught us.
I always liked the realistic imprint coming from a mediated support, the new sonic landmarks and the emblematic sounds of audio compression and the limits of some new media devices; it’s like collecting field recordings in a digital world but recorded by non-professional devices such as smartphones, dash-cams, live recorded concerts, cracks, crashes, impacts, then reinforced with classic sound design process. All of this is mediated by audio compression and so on, ending up building a totally different soundscape. The question in this case is: we’re immersed in a virtual multiverse since the ‘90s, so what has become of our listening and hearing process from a cognitive and perceptual point of view?
In the first iteration there was a very strong musical presence, since it’s more than a couple of years that I’m working on the integration of real and AI generated strings, especially cellos, and my first thought was to go in that direction.
What you hear is basically the sound of disaster, something close to white noise on a liminal level but full of patterns your brain can hook up to.
In this case, I decided to use AI systems to produce a sort of massive but realistic sonic avalanche instead of composing music.
By analyzing some frames of the videos using Llava, I was able to generate a prompt to feed a Latent Diffusion Model for audio generation and enrich my AI-Foley Sound Library which was built around 2021 with the Latentscape project using SampleRNN at the time.
“What you hear is basically the sound of disaster, something close to white noise on a liminal level but full of patterns your brain can hook up.”
I came to this conclusion after seeing the workshop Martina Carbone and Daniele Imani Nobar, which also works with me as assistant, held for Re:Humanism. They focused on using AI for Foley in a classical sense, generating and designing sounds for silent movies.
So after browsing my Sound Library and after listening to it repeatedly I then decided to only focus on these massive realistic sounds, following the morphings and the behavior of the images, while pushing the musical layer in the background, beneath all the other sounds.
At the same time the sound design workflow for an artwork series is very different from the one I use for my music or audiovisual performances.
For this reason, Mattia Magionami‘s support has become essential in the last year. I usually come up with a general workflow, a well defined sound aesthetics and an overall idea for some details, I make some starter templates and collect or generate materials. Mattia then takes charge of all the creative editing, synching, mixing and mastering of the final piece.
Franz Rosati. DATALAKE:GROUNDTRUTH N2, 2024
Another mesmerizing aspect of these animations is the fact that they are constantly morphing, without end, every disastrous event merging into another. The scenes therefore are captivating in presenting a thrilling spectacle of chaos, but at the same time, since the disaster is not permanent, but part of a flow, it does not seem like a tragedy. What have you found interesting in the technique of morphing and constant transformation?
I always worked with chaos intended as a high number of complex events happening together. This definitely depends on my fondness for musicians like Iannis Xenakis or Zbigniew Karkowski, Masami Akita or Paul Dolden, just to cite a few who had the first relevant impact with contemporary culture in my teenage years.
Chaos is potentially where you can create the most interesting connections, a pool of everything happening simultaneously, to be explored and where to find unexpected patterns and relationships.
When we talk about AI, learning is one of the key terms, but remembering is never mentioned. I feel that learning and remembering relate to a different sensibility rather than applying the same exact mechanism. The ease with which mass media and people tend to forget things is a crucial issue in our reaction to information overflow.
The constant morphing flow of images and events, forces you to check for patterns, to hang on landmarks in the screen space and try to learn what you’re seeing and remember something to not get lost in the flux and start to decode what is happening.
“Chaos is potentially where you can create the most interesting connections, a pool of everything happening simultaneously, to be explored and where to find unexpected patterns and relationships.”
Going back to the theme of disaster, this continuous morphing reminds me of that specific instantaneous feeling you have while the disaster is unfolding: a fall from the bicycle, or something more tragic, for instance, not necessarily physical.
In that moment your brain freezes the time: the unexpected event leads to a condition where it’s impossible to find patterns, and time stretches to infinity until this perception ends and you go back to a new, a new different reality to be re-semantized.
Franz Rosati. DATALAKE:GROUNDTRUTH N3, 2024
Both the images and the soundtrack have been generated using AI models. Can you explain the process of creation, and the differences between generating images and generating sound with AI? Given that you already worked with generative algorithms, what do these artificial intelligence programs contribute to your creative process?
I don’t want to go deep into technical stuff about workflows and code. I’ll focus on the creative experience with Generative AI compared to “old school” Generative Algorithms.
There are very huge differences in some aspects. We’re confronting with the same kind of approach at the foundation, based on giving rules and observing narrow variations as results, but we’re dealing with different kinds of complexity.
Dealing with Generative AI means we have to handle visual data condensed in a Latent space and then used to represent words we can prompt. It’s like a magic spell, and numbers are hidden behind this, since the final users have very few numerical parameters.
In “old school” generative process, we’re dealing with numbers and parameters in a very Pythagorean way, using equations, working with euclidean spaces, coordinates or controlling emerging features and systems behaviors.
In a creative approach, this leads the artist and the creative, in a very peculiar position. At the same time this technology is evolving so rapidly that is very much subject to paradigm shifts.
“When we talk about AI, learning is one of the key terms, but remembering is never mentioned. The ease with which mass media and people tend to forget things is a crucial issue in our reaction to information overflow.”
In this series you introduce the concept of “ground truth”, which is used in statistical models, cartography, meteorology, and the military to refer to information that is known to be true. Evidently, the fictions created by 3D simulations and AI models are putting into question what we can call “ground truth.” What has led you to work with this concept in this latest series?
Ground Truth is a technical term used in AI as well as Geographical Information Systems or Remote Sensing (which I used a lot in projects such as Distantia from 2023) and Statistics in general, used to prove if something is real based on empirical evidence.
It’s basically the opposite concept of Inference, which is a more common and popular term in the field of AI.
At the same time, if you ignore the meaning of the term, it could sounds a bit ambiguous: the word “truth” is very central since the advent of the internet and the alleged democratization of the media, this word is being used to represent and declare the state of declaring a single truth itself especially by conspiracy theory influencers, Q-Anon militants and alt-right propaganda at the point that we are now used to identify the word “truth” as a sort of red-flag…for fake, dangerous bad stuff and in general to deny the existence of many truths. So “Truth” is probably the word that best represents the gray areas of our time.
“«Truth» is probably the word that best represents the gray areas of our time”
This exact dichotomy and hard contrast led me to go for his title, to exactly evoke that kind of broken and twisted information system, depicting the distorted infosphere, the flow we are exposed to every day, in contrast with the quantum conception of reality, which gives us the tools to see the many truths reality is made of, that is instead becoming more and more strongly established.
