Through the Looking Glass

We live in a world where 3-D and computer graphics are an integral part of science, medicine, communication, and entertainment.

Today’s chemists create 3-D models of complex molecules, using them to discover properties impossible 25 years ago. Oncologists use 3-D imagery to precisely target tumors for radiation therapy. Architects have access to software that projects plans in three dimensions, allowing homebuyers to virtually walk through and see their dream homes before a building permit is even issued.

In movie theaters, the visual distinction between real and computer-generated images has vanished. Virtual actors, animals, and machines interact seamlessly with real life counterparts, and 3-D projection is commonplace. An entire new industry has emerged to create immersive virtual environments for both gaming and training.

And our world is on the cusp of being transformed by the ability to create 360-degree virtual reality environments using devices as ubiquitous as our smart phones.

In 2014, Facebook announced it would pay $2 billion to purchase the startup virtual reality company Oculus VR. One month later, Oculus Chief Scientist Michael Abrash visited UNC to speak to faculty and students about the revolution in technology and communication that 3-D imaging and virtual reality would bring. And he let the audience in on something few people outside the industry know: Technology that underpins much of today’s computer graphics and 3-D imaging has its roots at UNC-Chapel Hill.

A software engineering icon

In 1965, Fred Brooks sat in awe as he listened to “The Ultimate Display” — Ivan Sutherland’s influential speech that asked computer scientists not to see a computer as a screen, but as a window into a world. His riveting words described a technology that would change the very makeup of the room it occupied on command — imagine a computer that could produce a chair the user can sit in or handcuffs to hold a person in the room. “With appropriate programming such a display could literally be the Wonderland into which Alice walked,” Sutherland said.

“I knew right then that I wanted to work toward that goal,” Brooks says. Not long before Sutherland’s speech, he had left IBM to found the computer science department at UNC. Today, UNC’s Department of Computer Science is a leader in the field and a key part of North Carolina’s technological history.

At the time of the speech, Brooks was already a distinguished computer scientist. At IBM, he helped develop the revolutionary System 360 project. This was the first family of computers with applications that could work on any of the systems in the family no matter the size or speed. They were also the first that could type in lowercase. Ronald Reagan awarded Brooks with the National Medal of Technology for this work in 1984.

Brooks also played a large role in persuading IBM to build a new research facility in North Carolina — a move that is said to have sparked the growth of Research Triangle Park. His 1975 book, “The Mythical Man Month,” is often called “the bible of software engineering,” and his principles, designs, and innovations continue to influence computer scientists to this day.

In 1963, UNC offered Brooks a position as head of the computations center, which oversaw the use of a large computer in the math department. As part of his interview, he gave a lecture to several chairs within science and math departments titled, “Ten Research Problems in Computer Science.” One of the problems, according to Brooks, was that computer science research was extremely limited.

At the time, no one at UNC studied computer science — in fact, there was no free-standing undergraduate degree for it until 2001 (before then, students majored in mathematics with a computer science concentration). Brooks’ lecture intrigued several of the faculty enough to consider creating a department at UNC. In 1964, they asked Brooks to lead the new Department of Information Science, which would become the Department of Computer Science in 1971. “It was the hardest decision I ever made, but it became clear in a variety of ways that was where the Lord wanted me, so here I am, 52 years later,” Brooks says.

Funding was limited at first, and Brooks didn’t anticipate the department ever becoming very large, so he devised a strategy for it to make a name for itself. Rather than try to cover every topic under computer science, it would focus on just a few areas, which he named the “Pillars of Excellence.”

One of these pillars was computer graphics.

Picking up the pace

“Graphics had the tremendous advantage of not being respectable,” Brooks says, laughing. Both engineers and mathematicians considered graphics to be outside of their respective fields. Brooks, on the other hand, saw graphics as an interactive, 3-D tool to increase the productivity of scientists and engineers — to help users better visualize and understand complex structures, seeing things they wouldn’t be able to see without computer aids.

Early on, he asked the university provost for a list of researchers who would benefit most from such aids. One name caught his eye: Jan Hermans, Jr., a protein chemist studying how proteins are organized in a 3-dimensional space.

Hermans had been using wire models for his dissertation, as was the standard for the day. But by 1977, using a newly acquired graphics system from IBM, Brooks, Hermans, and researchers from Duke were able — for the first time in the world — to predict and generate a 3-D computer model of a protein.

For the next 30 years, chemists from around the continent came to UNC to study proteins on the graphics system. The department soon recognized the value in collaborating with the best minds in a given field, even if that field had little to do with the narrow definition of computer science that was then prevalent. “One of the best ways to do good computer science is to collaborate with really good people,” Brooks says. “That’s been a philosophy of the department ever since.”

Protein research needed visual displays, not necessarily realistic ones, but computer science researchers like Henry Fuchs still dreamed of creating truly immersive virtual environments. In 1978, Fuchs came to UNC with a history of modeling chromosomes on graphics systems and constructing 3-D models from laser scans of people and objects.

“I found kindred spirits here immediately,” Fuchs says. He, Brooks, and Steve Pizer — a researcher developing medical imaging — discussed the possibility of obtaining a mini computer powerful enough to run the programs they were developing. Until that time, it was standard practice for computer scientists to write programs on a set of cards and submit those to a central computer.

This need for speed led to the development of a technology called Pixel Planes at UNC in 1980. The new approach for designing graphics hardware allocated one processor per pixel on screen — meaning that many parts of the images on screen were generated simultaneously, vastly improving the speed at which graphics could be produced. Work on Pixel Planes continued at UNC through 1997, when the final iteration of the project, Pixel Flow, was developed.

The Pixel Planes technology was revolutionary. It established a platform from which much of today’s computer graphics and 3-D technology emerged. “Many people in the industry now say if you look at the insides of graphics processing units, many of the ideas started out with the Pixel Planes series of machines,” Fuchs says.

Navigating the ultimate display

Building a true virtual environment requires much more than just graphics and, today, UNC computer scientists are tackling the task of making these environments more immersive. Professors Ming Lin and Dinesh Manocha have extensively researched how to make the sound in these environments realistic. Lin has also focused on the physics of complex materials like liquids, clothes, and hair, and how to make them more lifelike.

Research like that opens doors for students — doors to jobs that could not have been imagined when Brooks and Fuchs began their work. Kelly Ward, one of Lin’s PhD students, was hired by Disney to do the computer modeling of Rapunzel’s hair for the animated hit movie “Tangled.”

Researchers at UNC are also exploring how to make these animations feel real, using a mechanical simulation called force feedback, which recreates the sense of touch by applying vibrations or motions to the user in a virtual environment. At UNC, researchers have accomplished this through a drug docking simulation using a real, mechanical arm. Biochemistry students feel the arm push back as they try to find where the drug could bind to the molecule on screen, and then feel it slide neatly into place upon success. With force feedback turned on, they can find the right spot on the molecule in half the time it takes without the technology.

Now, Fuchs says, much of the department’s work focuses on decreasing something called latency — the time between user input into a virtual environment and the environment’s reaction to it. When a user with a virtual reality (VR) headset turns his head, the software has to generate new parts of the simulation based on where the user is looking. Lower latency means this will happen more quickly, and consequently look more realistic.

Fuchs’ recent work has focused on virtual reality problems like this. One of his systems helped solve some of the latency issues in VR headsets by improving the speed at which a VR system tracks a user’s eyes. Like a ship captain uses stars to navigate the sea, Fuchs’ system uses hundreds of points on the ceiling to quickly find the user’s line of sight.

The computer science department continues to be at the forefront of virtual reality software and is one of the few places that develops “everything in VR from soup to nuts,” Fuchs says. At IEEE VR 2016, the premier international conference and exhibition for VR, UNC took home both the “Best Paper Award” and “Honorable Mention for Best Paper.”

Continuing the legacy

Today, some of the most tangible benefits of UNC’s computer science research are found in collaborations with medical science. In 1974, Pizer founded the Medical Image Display and Analysis Group (MIDAG), an interdisciplinary team that uses medical imaging to improve treatments. That early collaboration led to the formation of the Biomedical Research Imaging Center in the School of Medicine. Now, Pizer and his team use virtual environments to improve targeting in radiation cancer treatments — exposing more of the tumor to radiation while keeping the healthy tissue radiation free.

A substantial amount of VR development is currently underway at UNC, including work on a VR headset and a telecollaboration room in collaboration with NVidia, an industry leader in graphics cards. Additonal UNC industry partnerships include research done with Microsoft, IBM, the U.S. Department of Defense, and Intel. Relationships like these have led to similar companies setting up shop in North Carolina. Today, Google has an office on Franklin Street and Microsoft has one at Research Triangle Park.

Members of UNC’s Department of Computer Science have also launched several startups including InnerOptic (co-founded by Fuchs), which develops real-time visualizations for minimally invasive surgery, and a spinoff medical imaging software company called Morphormics (co-founded by Pizer). Another business, Impulsonic, uses high-performance graphics and acoustics simulations to provide physics-based sound effects for movies, video games, and live theatre.

The computer science department has developed remarkable products over the years, but Brooks stresses the role the department’s graduates play in taking what they have learned and applying it to technologies that are transforming the world. “The ideas get carried away in people,” Brooks says. “You publish papers and they change some things, but it’s the people that change most things.”

What Brooks says is certainly true. The video game industry, for example, traces its roots to the innovative minds working in virtual reality research at UNC in the 1990s. Graduates like Mike Capps — who served as president of the Cary-based Epic Games (creator of the Unreal Engine) for 10 years — have helped the area make a name for itself. Today, RTP houses more than 45 gaming companies and is the East Coast hub for the digital entertainment industry. In 2014, North Carolina’s video game industry brought $102.5 million to the state’s economy, according to the Entertainment Software Association.

Other accomplished graduates include Theodore Kim and Lawrence Kesteloot — who both won Academy Awards for their work on special effects software for movies — and NVidia Research co-founder and Vice President of Research David Luebke. Other graduates have gone to work for Disney, Microsoft, Google, and Apple, as well as a handful of esteemed universities including MIT and Stanford. These graduates are living testaments to Brooks’ philosophy that a measure to a department’s success comes from the impact its students have on the world.