There is a certain sensation in listening to the sound of the orchestra from within the orchestra itself: a physical, larger-than-life experience that disappears as soon as the sound is heard from the more distant perspective of the audience. The “Inside MPhil” project, a collaboration between the Munich Philharmonic Orchestra and Sofilab, aims to make this experience, previously reserved for the orchestra members themselves, accessible to the public.
With this focus on accessibility, we developed an interactive GPS-driven app that runs on standard smartphones with standard headphones to deliver a 6 DOF experience built on the Unity game engine with Dear Reality Binauralization technology to create an imaginary orchestra in a green field that can be sonically explored by walking inside. Special attention was paid to making the musical quality as vivid as possible, using a specially miked concert recording rather than an instrument-separated studio recording.
In this article, we describe the workflow, discuss the decisions we had to make along the way, and go over our learnings that we will apply to our next projects.
Introduction
Smartphones and headphones are ubiquitous. Using those devices available to anyone, the Munich-based sound art and design studio Sofilab conducts experiments which seek to unlock unprecedented possibilities for novel exciting musical experiences.
With the immersive audio app “The Planets”, artistically concieved by Gunter Pretzel and Mathis Nitschke and technically realized by Michael Hartung (née Klier), you can experience the famous eponymous orchestral suite by Gustav Holst, played by the Munich Philharmonic Orchestra, as an interactive audio walk in a park near you, increasingly all over the world.
How this started
In 2015, as an opera composer, Mathis Nitschke experimented with the idea of creating a music theatrical piece along the Munich river Isar („Vergehen“), which you experience by walking it. When the VR revival hit in 2017, he immediately started trying out different recording techniques to explore the question of whether it would be sonically interesting – and also musically satisfying – to take the listener’s perspective from inside the orchestra.
Video: Binaural field studies inside the orchestra – the Munich Philharmonic plays John Williams – Use Headphones!
The artistic co-creator of “The Planets” Gunter Pretzel already had this inside perspective of the orchestra: he was a violist with the Munich Philharmonic for over 30 years. He describes this experience inside the orchestra as so exciting and powerful that he, and many of his colleagues, have difficulties feeling the same excitement when they take the standard listener’s perspective in the concert hall. That’s why, over the course of his career, he has developed a number of projects that allow listeners to get intimately close to the musicians.
After Pretzel tried out Nitschke’s soundwalk “Vergehen”, he approached him with the idea to use smartphones and headphones to create this perspective, not knowing that Nitschke was already experimenting with this. As a proof of concept, they created the app “Inside MPhil” in 2019 which allows one to walk through the imaginary orchestra, set on a meadow. With this, you can experience one movement out of Schumann’s “Frühlingssymphony”, in high spatial resolution. You enter the orchestra acoustically and hear the instruments as if you were standing next to them, just as the musicians hear themselves and their colleagues: the violins, the oboe, the timpani, or whatever you want to hear.
The idea of letting the listener explore the musical sound field inside an orchestra is not new. However, we think that many experiments focus too much on the technical aspects. For example, many projects choose to record one instrument at a time to eliminate crosstalk of the sound sources. While this leads to good separation and localization of the individual instruments and prevents phasing issues introduced later by the binaural HRTF filters, the musical result lacks interest. After all, an orchestra must play as a whole, in a hall, in front of an audience!
So, for us, it was clear that we must use a live concert recording – and we will have to deal with all the technical compromises. But the music itself should remain uncompromised at all costs. If the music doesn’t work, nothing works!
The “Inside MPhil” app, effectively a GPS-controlled soundwalk, showed, as a successful proof of concept, that it is indeed possible to create a satisfying binaural interactive listening experience based on a live concert recording. As a result, a full-length symphonic experience was the next logical step, taking advantage of the learnings.
A full symphonic experience
“The Planets”, Op. 32, written between 1914 and 1917 by Gustav Holst (1874-1934) is a perfect balancing act between popular and sophisticated art music and thus became a model for much of the later Hollywood film music. Moreover, it is a work that challenges and showcases the qualities of an orchestra, its soloists and its conductor. All this made “The Planets” the ideal vehicle for the next edition of Inside MPhil.
“Mars” performed by the Munich Philharmonic Orchestra conducted by Krzysztof Urbański.
The original idea was to make the experience playful with an interactive spaceship flight to the planets, thus spanning the individual episodes, the landings on the planets and finally the original recordings. However, it soon turned out that the presence of the music was so powerful that it could not be incorporated into a game idea of any kind. But it also became increasingly clear that the music was capable of carrying the experience by itself. The task now was to lead an unbiased listener into this music unobtrusively but with increasing suggestiveness.
The format of a loosely joined narrative developed, in whose formulations the appearance and character of the planets, on the one hand, and those of the sounds and motifs of the music, on the other, were interwoven. An outer path was created, which led from the nearest planets to the most distant, and at the same time an inner path deeper and deeper into the music. The immediate real environment recedes more and more. And the reached final “distance” is at the same time the vastness of the universe as well as the immediate and mysterious nearness of the music in ourselves.
In order to make this process not too heavyweight, there are occasional refractions and double levels of imagination like for example the talking spaceship, the passing moon of the Jupiter, the pattering of the asteroids of the Saturn or the dangerous seductive power of the Venus.
The respective landing on the planets always becomes an impressive event. These landings are prepared by a careful dramaturgy of elaborately designed sounds, which should guide the user and at the same time get him in the right mood for the coming planet. Once he has landed, the original music comes into its own all the more strongly and unfolds with the full force of its artistic energy.
Inventing a visual layer
Originally, we envisioned the smartphone screen to be black and the phone to be pocketable so that the focus could remain entirely on the sound and the environment.
However, even after intensive research and development efforts, we have not found a way to reliably use the compass when the phone is not held in the default position. We use the compass to calculate the direction of the sound sources to the listener. If the vector of the phone to the listener’s body is not known, the listener’s relationship to the 3D sound environment is broken and immersion breaks down.
So we were faced with the task of developing a visual layer that was attractive enough to hold the phone throughout the listening experience, and at the same time not so attractive that it distracted the listener’s attention away from the sound and the environment.
Together with visual artist and programmer Luciano Pinna, we have developed a mechanism that transforms the current ground map into a star map: Paths and trails become Milky Ways, pulsars fly around, stardust fills the dark space.
The technical details
“The Planets” is created in the 3D game engine Unity, with the help of the commercial Dear Reality add-on.
The sonic heart of the app is Dear Reality’s Spatial Sound algorithm. The synthetic binauralization locates sound sources in the headphones coming from all directions of the three-dimensional space around the head. (In other words, you can not only control whether you hear the sound from the right or left, but also from the front or the back or up or down.)
The Munich Philharmonic records all their performances anyway, in their standard setup using main and spot microphones. Usually, they don’t use spot microphones on the brass as those are loud enough in the main mics. The orchestra’s recording producer Johannes Müller was extremely cooperative and provided not only the necessary spots on the brass but also went with Nitschke on some more experimental techniques like a quad outrigger setup which in the end he didn’t use, though.
In the first “Inside MPhil” app we took all these spot microphones as 3D sound sources and distributed them geographically on the map (the meadow). As a walking listener one then moves through these spot microphones. In the end, the user’s smartphone plays back a total of 22 tracks in real-time, all processed through HRTF filters.
A big challenge is to keep CPU cost under control. The Holst orchestra is double the size than the Schumann, and on top of that, we talk about a full symphonic experience of 1-hour in length. 40 simultaneously playing tracks, individually spatialized, are not feasible at all for even the most powerful smartphones.
A low-threshold offer with high artistic standards
“The Planets” is a soundwalk experience whose precise dramaturgical and technical design evokes a subtle but impressive artistic process. This way, music becomes an experience that cannot replace a concert hall but is on a par with it in terms of experience intensity.
This app represents the appropriate contribution of the “Munich Philharmonic Orchestra”, an institution of international high culture, to musical life outside the established spaces. And just as the successful balancing act of this music was mentioned at the beginning, this presentation is now also an example of the unification of a low-threshold offer with high artistic standards.
Go to https://the-planets.art/ for info on available locations and links to the App stores.
