Virtual Explorations

Newhouse communications professor Frank Biocca wrote the book—literally—on virtual reality with the publication of “Communication in the Age of Virtual Reality” (Routledge) in 1995. At that time, most people still viewed the technology as the stuff of science fiction. Today, Biocca is director of Newhouse’s Media, Interface, and Network Design Lab, known as the M.I.N.D. Lab, part of an international network of human computer interaction labs. At the M.I.N.D. Lab, what was once a thing of the future has become a main research focus, as the rapidly developing medium continues to grow.

Virtual reality (VR) technology allows users to interact with computer-generated simulations of images or environments in a realistic, nearly true-to-life manner through the use of a specialized headset. Similarly, augmented reality (AR) introduces computer-generated effects, such as video, audio, or graphics, but presents them in a real-world setting (think Pokémon Go). They are often referred to collectively as immersive media. “This is an entirely new stage of media,” says Dan Pacheco, Peter A. Horvitz Endowed Chair in Journalism Innovation at the Newhouse School. “Much in the way that television was a new thing, this is a brand-new medium—some people call it the ‘fifth medium.’”

Pacheco has been working with immersive media for several years. In 2014, he and then-student Irfan Uraizee ’15 spent the summer at the Des Moines Register in Iowa, participating in the groundbreaking virtual reality storytelling project “Harvest of Change,” produced through Gannett Digital. The project, which tells the story of a changing family farm in Page County, Iowa, through 3D game interaction and 360-degree video, went on to win a National Edward R. Murrow Award, which honors outstanding achievements in digital journalism.

Pacheco had also experienced the innovative immersive journalism projects “Hunger in Los Angeles” and “Project Syria,” developed by VR pioneer Nonny de la Peña, and he could see VR’s powerful potential as a storytelling tool. In spring 2015, he launched Virtual Reality Storytelling, the first course of its kind at any journalism school. Though he was initially worried about a lack of student interest, the class filled in the first three days of registration. This spring, it filled in three hours.

The interdisciplinary class is based in Newhouse’s Alan Gerry Center for Media Innovation, a creative hub where students from across disciplines work with next-generation technology. Pacheco’s students explore the differences between immersive media and traditional media—including possible health or ethical considerations that might accompany this new form of storytelling—and analyze existing immersive media packages created by Vive and Oculus developers. At the same time, they gain hands-on skills in 3D modeling, 360-degree video, and computer-generated imagery, creating immersive stories that they promote on Newhouse’s Interactive Media Wall.

Ultimately, students gain experience creating stories for various types of communications, including entertainment, journalism, advertising and public relations. “We’re at a point now where every serious media company is doing something in this area,” Pacheco says. “They’re asking us, ‘Do you have qualified students who are interested in interning?’ So we’re at this really fun initial time.” Pacheco’s students have produced stories on topics ranging from the extinction of dinosaurs to color blindness to the Dakota Access pipeline.

Carly Port ’17, a television, radio and film major, created a story package focusing on present day Auschwitz. Port had visited the concentration camp in 2012 and wanted to recreate the powerful experience for people who will never see it in person. Her story takes viewers on a virtual tour of Auschwitz II Birkenau as it looks today, based in part on photos she took during her visit. The tour is overlaid with audio from an interview with Holocaust survivor Israel Arbeiter, originally recorded by the World War II Foundation. Port had met and heard from a survivor on her own visit to the camp, a profound experience that she wanted to share with her audience.

Her project touched on what many see as the most powerful potential for VR storytelling—the ability to create empathy. “I believe VR has the power to inspire empathy unlike any other storytelling platform, and I wanted those who will not be fortunate enough to physically visit this historic site and feel all the emotions I felt to have this virtual experience,” she says. “I want to tell the stories of survivors, who have so much insight into a world we never knew.” Port says she would like to continue to work with VR and social justice topics after she graduates this spring.

Cognitive Impact

We might not yet understand how to harness and responsibly manage the power of immersive media, but the work being done at the M.I.N.D. Lab is moving us toward that understanding. Newhouse communications professor Makana Chock is an expert in media psychology who conducts research on media processes and effects. Chock has begun to explore the impact of virtual and augmented reality stories on the human brain. “This is a relatively new, under-explored area,” she says.

She suspects the effects of immersive media may be substantially different from those of traditional media, particularly in terms of memory and perceived realism—a measure of how realistic users judge media content to be. Studies have found that people will sometimes misremember events from traditional media as real-life experiences. “Is that stronger in a completely immersive environment? That’s what we’re trying to figure out,” she says.

She suspects the effects of immersive media may be substantially different from those of traditional media, particularly in terms of memory and perceived realism—a measure of how realistic users judge media content to be. Studies have found that people will sometimes misremember events from traditional media as real-life experiences. “Is that stronger in a completely immersive environment? That’s what we’re trying to figure out,” she says.

Chock brings people into the lab to experience immersive stories and scans their brains to measure which parts of the brain are responding. “We have old brains—our brains developed long before we had TV sets or movies—so they respond to stimuli to a great extent as if [the stimuli] were real, regardless of whether or not they are,” she says. This holds true even with traditional media; the fact that immersive media may have the same effect but magnified is central to its potential impact. “In VR your senses are telling you this is real—so how do you parse that out?” she says.

Chock is also using immersive media to take a deeper look at confirmation bias, which holds that people seek out or interpret information based on their preconceived notions, ultimately leading to errors in recollection and interpretation. In the lab, subjects are pre-screened for biases before experiencing a virtual reality scenario involving four men of different races at an airport. Chock measures subjects’ physiological responses both as they observe the scene and later, as they recall it. One theory is that immersive media might be used as a tool for changing attitudes.

There is currently no code of ethics in place to help guide producers of this kind of content, Chock notes. Some organizations, like The New York Times, use existing journalistic standards when producing immersive stories, but without truly knowing the effects of this kind of content, it’s hard to know if those standards are enough, she says.

The University is the ideal place to ask these kinds of questions, she says, hoping to see a center established at the Newhouse School that would advance education, training and research in immersive media. “We have student storytellers creating content, we have faculty studying the effects of the content, we have the M.I.N.D. Lab, where we can measure these things in real time, and we have industry connections to get the information into the field,” Chock says.

Understanding the Ramifications

Of course, immersive media technology will continue to take off, at the academy as well as in the industry, even before its ramifications are fully understood. While it is no longer a thing of the future, its full potential still lies in the future. Learning how and where best to use it, as well as using it with care and according to an agreed-upon set of standards, will be key to the further development and success of VR and AR.

“These are a whole new set of powerful capabilities that can be used for good, but also for scary things,” Pacheco says. “The fact that you can trick someone’s brain into thinking that they are somewhere else, that they are someone else…that can be really powerful.