What should we do to record more realistic 360° panoramic videos? Increase the resolution? What can we do besides that?
The first time I watched 360° panoramic videos, I thought it was “flat.” One of the reasons should be in the sound of the video. You can change the viewpoint in these videos, but it doesn’t make any changes in sound localization.
3D games already have 3D audio effect. Why don’t we use the 3D audio with 360°panoramic videos? Keeping this simple idea in mind, we developed a multi-channel surround microphone for 360° panoramic video recorder(THETA by RICOH) and original video player with 3D audio.
This microphone, named “Ambisonics360 Microphone,” consists of 4 unidirectional condenser microphones which are directed to vertex of the tetrahedron. Record the video and 4 channel sound with portable recorder, at the same time. When you play them, our custom video player mixes 4 audio channels into stereo depending on your viewpoint.
This system is based on a surround sound called “Ambisonics” in the 1970s. Ambisonics is different from common formats like 5.1ch, and it doesn’t depend on a particular placement of speakers. It has an advantage in that the sound will be re-synthesized from the recorded sounds to suit the playback environment. However, Ambisonics been forgotten for decades because of a limitation of the technologies and the little demand at that time.
Nowadays VR technologies — like Oculus Rift — are in the news almost everyday, but most of that are for the visual representations. Most of these new VR technologies are meant for the visual representations. Our challenge here is to shed light on the importance of the auditory sense.
Street noise of hometown, chimes of your school … these sounds bring your old memories back and make you feel like you were there. We believe that sounds are, at least, as important as visuals in order to provide realistic VR experiences.
“Seeing” is Believing. What about “Hearing”?
When you are recording with Ambisonics360 Microphone in a situation like this; there is a barking dog, and on the other side a car is driving past you. If you played the data — the sound and video recorded in that situation — with a head mounted display or our original player, you can feel like as if you were in that place. While you see the dog, you can hear the a bark from the front. If you changed the viewpoint to the car, you hear the sound from right to left (or left to right) like actually the car drove past you.
Inside Ambisonics360 Microphone, we have 4 unidirectional condenser microphones which are directed to different directions each other. So we can record 4 track at the same time and play them on our original player, you can hear the mixed sound depending on your viewpoint.
Nao Tokui (Qosmo, Inc.)
Miyu Hosoi (Qosmo, Inc.)