VR and AR: The Art of Immersive Storytelling and Journalism

Storytelling traces its roots back to the very beginning of human experience. It's found its way through multiple forms, from oral traditions to art, text, images, cinema, and multimedia formats on the web.

As we move into a world of immersive technologies, how will virtual and augmented reality transform storytelling? What roles will our institutions and students play as early explorers? In the traditional storytelling format, a narrative structure is presented to a listener, reader, or viewer. In virtual reality, in contrast, you're no longer the passive witness. As Chris Milk said, "In the future, you will be the character. The story will happen to you."

If the accepted rules of storytelling are undermined, we find ourselves with a remarkably creative opportunity no longer bound by the rectangular frame of traditional media.

We are in the earliest stages of virtual reality as an art form. The exploration and experimentation with immersive environments is so nascent that new terms have been proposed for immersive storytelling. Abigail Posner, the head of strategic planning at Google Zoo, said that it totally "shatters" the storytelling experience and refers to it as "storyliving." At the Tribeca Film Festival, immersive stories are termed "storyscapes."

The stories we have selected use both 360-degree video and virtual reality. The videos are captured with 360-degree cameras, while immersive computer-generated VR worlds are developed in game engines like Unity and Unreal. Viewers adopt different perspectives, sometimes even becoming protagonists who can walk into the story and interact with people and objects inside the virtual world. However, with advances in "light field" (which mimics the way you actually see the world) and motion-capture technologies, the future will bring volumetric avatars and photorealistic three-dimensional mixed reality spaces.

The Impact of Immersive Storytelling

The sense of presence in VR transforms the storytelling experience. Participants become part of an environment with an incentive to act and respond to the events they encounter. We see this in the early work of the virtual reality pioneer Nonny de la Peña at USC. Her groundbreaking Hunger in Los Angeles experience, which premiered at the Sundance Film Festival in January 2012, places you in the middle of a food bank line in LA (see figure 1). In her TED talk, de la Peña describes the first rule of VR design: Begin by thinking of your body in the space. The focus of VR design is not the camera frame, but the embodied visitor.

Built in Unity 3D with a body-tracking system and audio recordings of the actual event, the experience often triggers intensely emotional responses from the participants. Viewers walk into a virtual re-creation of the event, allowing them to experience and respond to it. The project was funded partially by USC's Annenberg School for Communication and Journalism and its MxR interaction lab. A more recent experience from 2016, Across the Line, places participants in the middle of anti-abortion protestors outside an abortion clinic. Blending journalism with storytelling, de la Peña refers to her work as "advocacy journalism."

Immersive Journalism and Storytelling

Researching the initiatives in this area, we're seeing a number of innovative student projects. At the University of California, Berkeley's Graduate School of Journalism, two students experimented with 360-degree video as way to explore the emerging medium and its potential to create empathy. Their master's project, The Wait [https://thewaitgermany.com/], explores the refugee crisis in Europe by focusing on the lives and challenges immigrants face in Germany waiting for asylum (see figure 2). This new-media project includes text, animation, interviews, and 360-degree video.

David Cohen, who teaches a new-media storytelling course at the school, encouraged the students to use 360-degree video for the way it can convey the individuality of human stories. The students interviewed nine people in five German cities, taking viewers into the kitchens and crowded apartments and onto the cold Berlin winter streets as asylum seekers wait for resources. The project invites you to enter their "tiny worlds" and step into their shoes to understand the refugee story.

When the students filmed their project in January 2016, there was little guidance on filming and editing 360-degree video; as part of the project, they wrote and shared best practices for their peers and the department. Beyond graduation, their project is part of the fast-evolving conversation around using the technologies of 360-degree video and VR for journalism.

John Hopkins University recently opened a new Immersive Storytelling & Emerging Technologies (ISET) concentration in its Film and Media Master of Arts program. Designed by the well-known VR/AR filmmaker Gabo Arora, a highly diverse group of students will focus on ways to use virtual and augmented reality along with artificial intelligence to create new forms of cinematic storytelling for the digital era. Roberto Busó-García, director of the Film and Media Master of Arts program, noted:

"With this new concentration and set of initiatives, we will create a space to explore some of the most pressing issues related to emerging technologies and immersive experiences as they relate to storytelling, artistic expression, and social impact.... We aim to continue bridging the access gap between talented and visionary individuals from all walks of life and the industry at large."

At The New School in New York City, students in the Parsons School of Design can minor in immersive storytelling. Through coursework, students explore experiential and immersive storytelling in virtual, augmented, and mixed reality; data visualization; 360-degree spherical film; and interactive theater. Students also can experiment with 3D in the Motion Capture Studio. Those in the Art, Media and Technology school work on projects that tell nonlinear stories, experimenting with games, virtual reality, and immersive theater. School of Media Studies courses explore new-media narratives, sensory experiences, and developments in transmedia.

In the Eugene Lang College of Liberal Arts, undergraduate students in the Rendering World course this spring will explore media history and theory and the way they inform new affordances and potentials of virtual reality. Students will investigate how world building in VR might inspire new historical and theoretical perspectives. With activities going on across schools, last fall The New School opened the XReality Center, a new research center to advance the study and design of emerging technologies in virtual, augmented, and mixed realities to serve all students and faculty across its five colleges (see figure 3).

The Mobile Virtual Reality Lab is a project funded by Florida International University's School of Communication and Journalism in collaboration with Eyes on the Rise. The community-based initiative is focused on creating digital story opportunities for south Florida communities that highlight the effects of the rise in sea levels (see figure 4).

Some VR projects featured the experience of rising sea levels, showing how it will impact the region's infrastructure. The project relies on a social media campaign to engage the larger community with mobile VR experiences to interact with the resources.

As part of the project, Digital Media Studies students from the Department of Journalism & Media presented their research on using VR to understand the medium's ability to examine environmental communication at the 2017 Conference for Undergraduate Research.

An interdisciplinary theatrical performance was also created. A Sea Change takes the audience on an multisensory and multimodal journey where students and faculty draw together on stage from science, journalism, history, architecture, arts, and the humanities, with the goal of having research share the stage with interpretative arts and drive action.

Immersive Live Theatre

Theater itself is become immersive. Shakespeare's Hamlet is now being adapted in virtual reality at the New York University Tandon School of Engineering. Led by Javier Molina, an adjunct assistant professor of Integrated Digital Media, a group of actors, artists, and developers have created To Be with Hamlet (see figure 5).

Remarkably, this version is one of the first performances in theater to be staged live in a multiuser virtual environment. The audience can be live onsite or participate remotely through HTC VIVE headsets. Motion-capture is used to track the actors' movements and is processed through iKinema's LiveAction platform to produce realistic digital characters (see figure 6). The audience is no longer passively watching the play but interacting within the virtual space.

As the Director, David Gochfeld, noted, "By immersing [the audience] in the setting — allowing them to virtually share Hamlet's world for a short time — we hope that they might gain a more visceral understanding of Hamlet's experience."

The Future of Immersive Storytelling

As Janet H. Murray, author of the now classic Hamlet on the Holodeck, said:

"Empathy in great literature or journalism comes from well-chosen and highly specific stories, insightful interpretation, and strong compositional skills within a mature medium of communication. A VR headset is not a mature medium — it is only a platform, and an unstable and uncomfortable one at that."

With immersive storytelling, we find ourselves at the birth of new medium. There is no roadmap, no tools, and no templates for the new storyforms we need to create. Students, faculty, artists, and filmmakers, both in and outside higher education, are experimenting with new solutions. And as Murray noted, "Most of what these adventurous folks (myself and my students included) are producing is terrible, which is just as it should be."

Failure is part of the learning process for immersive storytelling, as is bold experimentation. Storytelling does not embrace a new medium overnight; the conventions and structures develop incrementally through an ongoing conversation with cultures looking for new ways to understand themselves. Higher education can provide the space for this creative process.

As we move into this new era of immersive storytelling, it will impact a wide range of disciplines. These projects demonstrate the need for an interdisciplinary approach, bringing together subject experts, engineers, artists, and media specialists. In the end, it's not just about continuing the long tradition of storytelling but about giving our students the skills they need to tell their own stories and solve the pressing problems of the future.

---

Emory Craig is Director of eLearning and Instructional Technology at the College of New Rochelle and Co-Founder at Digital Bodies.

Maya Georgieva is Chief Innovation Officer at Digital Bodies.