Take a moment to focus on the sounds around you. Maybe you’re sitting at a desk and someone walks by in the hallway. Even though you can’t see them, you can hear their footsteps and can tell when they get closer, when they walk away, and that they’ve just passed by to your left.
We take sound cues from our environment every moment of every day and rely on them much more than we consciously realize. Virtual reality (VR) and gaming researchers are studying how to replicate the paths sound takes before reaching our ears — they call it three-dimensional spatialized sound. In order for a virtual world to be truly immersive, the sounds in that world need to give us these cues. That’s what engineers of 3-D sound simulation hope to achieve. Besides VR, these sound propagation technologies are also used for evaluating the acoustic effects of architectural buildings and noise modeling in both indoor and outdoor environments.
In 2011, PhD students Anish Chandak and Lakulish Antani approached computer science professors Ming Lin and Dinesh Manocha with the idea of using existing UNC technologies for physically based sound simulation to create a startup to develop sound simulation tools for different applications. This resulted in the formation of Impulsonic — a startup company supported by a Carolina Express License from the UNC Office of Commercialization and Economic Development, as well as strong backing from the Department of Computer Science, College of Arts & Sciences, and Launch Chapel Hill, a member of the Innovate Carolina network.
Starting a company without much business experience can be challenging, Manocha points out. “It is always a bit risky in the technology space because very few startups really make it.”
This past January, gaming giant Valve, widely known for its Steam platform, purchased Impulsonic. With the acquisition, Chandak and Antani now work at Valve on Steam Audio, a direct continuation of Impulsonic’s product, formerly called Phonon. “As far as our products and team is concerned, we are with Valve now,” Antani says. “Steam Audio is just the next release of our original product at Impulsonic.”
The acquisition will provide a vast amount of resources to help the team further develop one of the best 3-D sound simulation solutions into an even better product. It also allows Valve’s enormous audience, with more than 125 million Steam users, to experience this incredibly realistic sound software. “This is a great mechanism to transition the ideas from university research into widely used commercial products,” Manocha says.
Hearing in 3-D
In current games, sound is usually layered on top of the visuals and does not interact with the world on the screen. It’s similar to the soundtrack of a movie, which doesn’t change based on where the characters are in the film or what’s happening in it. Movies like “Apocalypse Now” and “Saving Private Ryan,” have attempted to change this by using stereo speaker systems that play sounds left or right, to make the viewer feel like they’re immersed in the movie. While that can be effective, 3-D audio does more than play sounds out of different speakers and actually manipulates sound waves so they act as they would in the real world.
Before it reaches our ears, sound waves undergo a number of changes based on the environment, the position of the sound, the location of the listener, and even the shape of the listener’s head. Phonon uses several algorithms to manipulate sound sources before they reach the listener to reflect these changes.
“In my office, there is a carpet and a wall,” Manocha describes. “Imagine if my floor was all concrete and my wall replaced by all glass. Do you know how my office would sound? There would be a lot of reverberation and less absorption.” Phonon can predict these sound effects without building the physical prototypes. “We are working with many engineering and architecture design firms to design better tools for acoustic evaluation,” Manocha adds.
Manipulating sound waves so they reflect this change is called physics-based sound propagation. It also takes into account the relative positions of the listener to the sound source and is likely the reason Valve chose Impulsonic over competing 3-D sound startups, according to Antani. “On the industry side, you can probably count the number of teams that are working on this on one hand,” he says. “It’s not always something that people work on within the virtual reality realm.”
Three-dimensional sound researchers typically spend their time studying the shape of the listener’s head to determine how sound molds itself. To do this, they use a mathematical model called HRTF (head-related transfer function). This is part of a larger area of sound research called binaural rendering, which attempts to produce a form of surround sound that’s heard not only from the left and right sides of the body but also above and below.
“If you are sitting to my left as I speak, some of my sound waves will go over what’s called line-of-sight sound to your left ear directly. Then, parts of those waves bend around your head and go into your right ear,” Manocha says, adding that the latter waves produce lower frequency sound effects. The human brain can process the differences between these sounds to figure out the exact direction of the original sound source.
In the same way prescription glasses or fingerprints are specific to an individual, HRTF varies for each listener. The current solution in games and virtual reality is to use a generic HRTF, but this means some people will hear sounds more realistically than those whose HRTF does not match as closely to the generic one. Antani and Chandrak chose to work more on sound propagation because of the relative ease of getting a decently accurate HRTF. “Once you get the HRTF to a reasonable baseline, it becomes super important to make sure that any sound you hear is coherent with whatever you are seeing,” Antani says.
Binaural rendering — and, by extension, Phonon — works best with headphones, because each side represents an ear in the sound simulation setup. With speakers, the sounds will play with the rendering, but the listener’s environment outside of the virtual world will affect how the sound reaches the ears and muddy the effect. Most virtual reality headsets come with headphones, so this is usually not an issue unless you’re listening to demos of binaurally rendered sound outside of VR.
Following a different path
The move to Valve was a natural transition for Chandak and Antani, since its structure actually resembles that of a startup. “Valve is very well-known for having a flat organizational structure, which means you don’t have bosses and such,” Antani says. “Everybody owns a piece of whatever it is they’re working on. That’s very much like a startup, except you have access to resources from a larger company.”
For technology startups, one of the best exit strategies is acquisition, according to Lin and Manocha. With the money, resources, and support of a successful company, Chandak and Antani can focus on developing the technology while leaving the bureaucratic issues to Valve. “As a CEO, you’re also worrying about the marketing and customer support, along with fundraising,” Manocha says. “It gets very challenging doing five jobs.”
In parallel to the commercial developments at Impulsonic, Lin and Manocha have continued their research in sound synthesis and propagation with other undergraduate and graduate students at Carolina. Some of their recent graduates are currently working on sound simulation research and commercial products at other leading companies such as Oculus/Facebook, Microsoft, and Google.
Lin and Manocha are very proud of their students and optimistic about the future of Phonon. With the acquisition, the likelihood that this UNC research will be adopted by the world of 3-D sound is high. “VR wasn’t hot in 2011, but Anish and Lakulish took the risk,” Manocha says. “They could have pursued high-paying industry jobs like their fellow graduate students, but instead took a much more nontraditional route. You have to give them a lot of credit for the success of Impulsonic.”