PlayStation 5 Lead System Architect Mark Senny Revealing more details about the Tempest Engine, these sounds make the sound appear to come from multiple directions in Sony's upcoming game console.
During the live broadcast, Cerny delved into the technical design of the console, which is expected to debut in late 2020. Sony has more powerful processing capabilities, is investing heavily in 3D audio through a dedicated Tempest Engine, and is using custom designs to make the sound more realistic and contextual.
The challenge is that players can listen to today's games in many different ways, including a variety of headphones, TV speakers or a surround sound system with up to seven speakers. Sony is optimized for headphones and still works on TV and surround sound systems.
With PlayStation 4, audio usually gets very little processing power available, even less than the SPU (Co-Processor Unit) core of the Sony PlayStation 3's Cell processor. In contrast, PSVR virtual reality headsets have 50 different sound sources. Tempest Engine has enough processing power (one kind of processing called single instruction, multiple data or SIMD), because all the eight core processing capabilities of PS4 have been combined-only for 3D audio.
But PS5's new Tempest Engine will support hundreds of sound sources, which means that sounds heard in the game may come from hundreds of different directions. When you come across things like raindrops, you hear them coming from many different directions. This will give you a sense of presence as if you were transported to another world.
You can also track locations or the location of objects more precisely. If an enemy sneaks into you, you will be better able to tell you. Such sound effects are common in today's games, but they are often confused. I remember playing Fortnite and heard footsteps in the house. But I don't know where these sounds came from, and I was ambushed while searching for a house.
One of the biggest challenges is that we all hear different voices. Our ear canals are different, so the path of sound into our ear drums is also different.
Audio researchers call the combination of sound and direction our way of hearing, and they can measure it with a head-related transfer function or HRTF. You can see an example in the image above. Everyone has a different HRTF, and you can take measurements in special labs. Some people may not recognize spatial audio at all.
Sony cannot do this, so it must provide a default HRTF and see how to modify it for everyone.
"We want everyone to have 3D audio," Cerny said.
Instead of using existing technologies such as Dolby Atmos, Sony decided to do its own 3D audio work. But it created the Tempest Engine in PS5 as a custom processing resource to ensure that 3D audio gets its due share in the overall processing in the system. That's expensive, but Sony hopes that bet will pay off.
"There are many changes, so it's great to have the computing resources of Tempest Engine," he said.
Using HRTF in games is very complicated because sounds seem to come from one direction or from general areas in real life. You can think of footsteps and explosions in the distance.
To handle a variety of sound devices, such as TVs and surround speakers, Sony is developing a product called Virtual Surround Sound, which gives you the best place to listen to sound on the sofa. The team is working to improve it so that you can sit wider, which is how engineers tried a few years ago to provide better best points for 3D viewing on TV.
But Cerny said that the sound of the PS5 is very different from the sound of the older PS4.
"I heard, wow, I feel different. The improvement is obvious," he said. "It seems that the sound comes from the real world, not the game."
Cerny said it will take years to achieve outstanding 3D audio fidelity in many games.
"This is a journey," he said.