Binaural audio synthesis in the world of Star Wars

In the famous Star Wars film released in 1977, a scene shows Luke Skywalker learning the art of Jedi combat from his mentor Obiwan Kenobi. Using his lightsaber, he must block the shots of a probe that is flying around him. At the end, Obiwan places a helmet on his padawan's head, making him blind and asking him to use "the Force". Matthieu Aussal, a research engineer at the Centre for Applied Mathematics (CMAP*), has developed a prototype of an immersive audio game in which you find yourself in the place of Luke Skywalker in this situation. Players, both sighted and blind, can immerse themselves in the Star Wars world, to confront this test. Visitors of the French national event “Fête de la Science” at Institut Polytechnique de Paris will be able to test it.

All you need is a smartphone and a headset with a sensor that detects which way you turn your head. You are then immersed in a natural soundscape with a river running in the background. But soon the enemy probe appears, making high-pitched beeps before firing at you. You have to draw your smartphone like a lightsaber and place it in the direction of the probe to block the shot. As the game progresses, the probe moves faster and faster, higher and higher, and shoots more frequently. The challenge is to reach the "Jedi Master" level.

Of course, this is not about feeling "the Force", but rather about understanding how we can construct a mental image of our environment, solely through our auditory system. The fact that we have two ears results in a time lag between the arrival of sound in one and the other, which gives us information about the direction of the sound sources. Furthermore, the morphology of our face distorts sounds a little: a sound coming from behind us does not have quite the same timbre as an identical sound coming from the front. We are not necessarily aware of these distortions, but our brains have learned to analyse them from an early age.

One of the challenges of "binaural" sound synthesis is to determine which filters to apply to a sound so that it appears to come from this or that direction, in order to trick the brain and create an immersive artificial environment. Jean-Christophe Messonier, from the Conservatoire National Supérieur de Musique et de Danse de Paris (CNSMDP), and Matthieu Aussal have developed a new way of conceiving these filters, which is more respectful of the initial timbres of sounds. "This gives the feeling of really hearing the sound of a lightsaber, for example, and thus gives coherence to the landscape that we want to create," he explains.

The game code was developed with the “castor” open-source C++ library, created by CMAP researchers. This library allows the rapid prototyping of industrial software. Tested in partnership with the "Science and Video Games" Chair of École Polytechnique and the association Mix-Handi Cap sur la Vie, the game is inclusive in the sense that blind people can take it in hand and play with sighted people. The sighted people who have tested it actually do better when they close their eyes! Finally, in the light of this research, could the Force be binaural listening?

*CMAP: a joint research unit CNRS, École Polytechnique - Institut Polytechnique de Paris