There is a certain sensation in listening to the sound of the orchestra from within the orchestra itself: a physical, larger-than-life experience that disappears as soon as the sound is heard from the more distant perspective of the audience. The “Inside MPhil” project, a collaboration between the Munich Philharmonic Orchestra and Sofilab, aims to make this experience, previously reserved for the orchestra members themselves, accessible to the public.

With this focus on accessibility, we developed an interactive GPS-driven app that runs on standard smartphones with standard headphones to deliver a 6 DOF experience built on the Unity game engine with Dear Reality Binauralization technology to create an imaginary orchestra in a green field that can be sonically explored by walking inside. Special attention was paid to making the musical quality as vivid as possible, using a specially miked concert recording rather than an instrument-separated studio recording.

In this article, we describe the workflow, discuss the decisions we had to make along the way, and go over our learnings that we will apply to our next projects.

“Inside MPhil: Frühling” is a unique virtual experience that allows users to immerse themselves in a live orchestra performance by the Munich Philharmonic. Using a GPS-enabled smartphone app, participants can walk through a virtual soundscape where they hear instruments as if standing next to the musicians. The app utilizes advanced spatial sound algorithms, offering a 360-degree auditory experience, bringing Schumann’s “Spring Symphony” to life in any open space.

Here’s Mathis in his own words on the birth of this project.

“Maya” is an innovative mixed-reality techno-opera set in the ruins of a former heating plant in Munich. This unique experience blends augmented reality, opera, and techno music, allowing audiences to explore the remnants of a past civilization through the eyes of “Maya,” a digital being on a mission to save humanity. The immersive performance used an AR app that guided participants through the site, creating a dynamic, interactive experience where the physical and virtual worlds merged seamlessly.

Mathis Nitschke held an artist talk at the Opera Beyond conference in Helsinki 2019 about the Mixed-Reality-Techno-Opera MAYA about the very essence of this project: “Transforming opera audiences into active explorers”.

You can watch the full presentation here.

Documentary film maker Felix Hentschel accompanied the production of the Mixed-Reality-Techno-Opera MAYA and held talks with the artists involved. The resulting film (ca. 40 min) illuminates the development of the Maya character and provides insights into the conception of the music and the design of the MAYA augmented reality app. Access the video in this blog post.