Mixed Reality: A Revolutionary Breakthrough in Teaching and Learning

Teaching and learning with technology, prior to this point, was mostly limited to supplementary collaboration tools for communication: learning management systems and electronic texts. The technology was simply an electronic aid to traditional didactic teaching, adding a little modern muscle to the same ways we have taught and learned for decades. We are now entering an era where technology really does the teaching, providing an entirely new mechanism for learning.

In November 2014, Case Western Reserve University (CWRU) and Cleveland Clinic broke ground on a new Health Education Campus (HEC), a 485,000-square-foot building named the Sheila and Eric Samson Pavilion, to house the university's schools of dental medicine, nursing, and medicine, including Cleveland Clinic's track within the medical school, the Lerner College of Medicine. The two institutions created this multischool campus to achieve two goals: to provide students the most advanced learning technology anywhere; and to prepare them to lead in a new era of team-based care. As part of those goals, the partners agreed that the HEC would not house cadaver labs but, instead, would identify new ways to provide anatomy education.

After teams from the two institutions investigated several technologies that didn't quite work as expected, the respective leaders of the hospital and the university received an invitation from Microsoft to view a then-secret technology still in development. The mixed reality device we now know as Microsoft HoloLens so impressed the two that their academic and technology leaders soon scheduled a follow-up visit.

I was very skeptical on the first trip to Redmond, Washington, but upon viewing several demonstrations quickly reversed position. In fact, one of my first comments was something like: "You just changed my life. I didn't expect to see this in my lifetime, let alone in my professional lifetime."

Several others who attended had similar responses. The ability to interact with 3D holograms—while also seeing instructors and classmates—provided extraordinary learning opportunities to advance the study of anatomy. Soon after the visits, the university, hospital, and Microsoft leaders agreed to develop an anatomy-focused HoloLens app that illustrated the educational potential of the approach.

CWRU identified magnetic resonance imaging pioneer Mark A. Griswold as the lead faculty member for the project. Professor Griswold recently had founded a new campus institute, the Interactive Commons (IC), which he describes as "a place and process to encourage radically different academic disciplines to begin new conversations, exchange knowledge and find new ways to look at complex problems—something that happens today more through happenstance or serendipity."

The IC did not yet have a physical home and included only one staff person. CWRU's Office of the President, the School of Medicine, and University Technology pooled funds to create the IC's innovative visualization space, hire and equip staff, and start creating the app later dubbed HoloAnatomy. At the same time, the team began developing and testing elements of what would become a full anatomy curriculum. Once Microsoft HoloLens became public in 2015 and Griswold appeared with the device at the company's annual BUILD conference, interest quickly mounted among faculty, university leaders, the business world, and higher education in general.

Visualization has long helped us understand the abstract by morphing various analyses into pictures. Viewing items in three dimensions helps us understand how they really appear; augmented, virtual, and mixed reality (AR/VR/MR) technology allows us to be in a visual space to help us understand various scenarios.

Technology is finally evolving that has the potential to revolutionize teaching and learning, research, entertainment, and information sharing for the individual. Devices such as the Microsoft HoloLens (for one) offer the ability to teach and learn in a mixed holographic reality. Users can see how the aortic valve in a heart actually works, collaborate with experts near and far, and see what others see—all in real time. Students can learn about geography not only by reading or watching a video but also through being immersed in it. Their brain believes they are there. They have an emotional response to this experience and they learn more holistically.

In MR, viewers are not limited to observing only what a video director wants people to see. Instead they can direct their vision in 360 degrees across an entire space and in every direction. They can walk through the streets of Rome, see a beating heart as blood flows through it, design and be inside architecture prior to finalizing building design, and more. In the past, we learned about historic settings like the Roman Colosseum by reading a textbook with a 2x2-inch pixilated picture. We had to imagine what the space looked like from the grounds in its center, the stands, or even outside. Today students can watch videos, see pictures, share experiences, and—with AR/VR/MR—also be totally immersed in the subject matter. They can listen to cafe patrons, fly over the Colosseum in a hot air balloon and get a sense of the geography, occupy Caesar's box seat as the Colosseum gets restored, and even give the thumbs up or down to one of the gladiators in a battle. Multiple students can work collaboratively on a holographic project, seeing and interacting from various vantage points. Yes, this is a gamer's dream. Beyond the gamers, though, there exists a new pedagogical paradigm. Educators have been saying for decades that the best way to learn is to experience something firsthand.

Think about the experiences that students can have with holograms and that they may not otherwise have—medicine and science, travel and geography, architecture and art, to name a few. These types of experiences will call for new teaching methods, new learning spaces (move the furniture out!), and new demands from students for the emotional response.

In March 2016, Microsoft formally released HoloLens to developers. CWRU's early relationship gave us access to the device, the developers' kit, and training materials much earlier, with the expectation of feedback prior to release. The device is a Windows 10 computer programmed in Unity, a ubiquitous platform for graphics applications. It provides tools for our artists and developers to design and create for HoloLens in 3D space. We chose Microsoft Azure cloud to host our development and storage. The typical R-1 university wireless environment is adequate to support the content delivery. So, the actual enterprise investment, after the HoloLens devices, was minimal. In June 2016, CWRU and Cleveland Clinic released HoloAnatomy, the first third-party app in the Microsoft HoloLens store.

Shortly thereafter, university artists began working with medical school faculty to create anatomically accurate visuals in 3D. Developers quickly learned the ins-and-outs of how the device worked. These components are not merely hardware and software; they are an entrée to imagining new paradigms for learning and collaboration. Students are now able to explore the wonders of human anatomy via a dynamic journey through the body. This digital model provides an opportunity for robust simulations of living tissues, including physiological and biochemical processes. It will enable them to extract or expand particular organs or systems, view the body from a range of angles, and enable exploration without the fear of making a mistake. As IC Executive Director Erin Henninger explains, the programs are designed "to shift from centuries of dissection and 2D illustrations to a 3D systems-level view, at true human scale."

[https://www.youtube.com/watch?v=SKpKlh1-en0]

Microsoft HoloLens enables this exploration to occur on a social level. Students are not closed off from the world or the people around them when they wear HoloLens. With this MR device, the transparent visor of the headset allows wearers to see and hear each other and see their "real" environment as they simultaneously interact with a digital anatomy model together as a group—just like they experience in a traditional lab setting.

Think of it: when students learn about human anatomy using a cadaver, they do not get the experience of looking at the living colors or textures, or seeing how organs function, or learning how blood actually flows. In addition, they are presented with only the disease or health factors of the body they are studying. If that cadaver had liver disease, the students would not experience how a healthy liver looks. Also, the cadaver is not available for the students' reference once the anatomy class is over.

With MR holographic learning, students work individually and collaboratively on 3D holograms to learn all of the parts of anatomy, and they are able to refer to these even after the class is finished. A full-sized transparent body appears before the viewer, arms outstretched at either side. Inside is a full skeleton. Then a circulation system appears, arteries and veins lacing through the body. Then the full musculature is revealed. Using HoloLens, students can explore—individually or together—whole biological systems. The full curriculum enables students to see systems not only separately and together but also in motion, allowing users to see how the heart moves and observe the heart valves opening and closing while hearing the sounds of a heartbeat. Students can see how the brain processes information from the body and how tumors in the brain affect body movements.

If students want to examine the arm to understand precisely how its muscles and bones work together, they can approach the limb more closely, look beneath the skin, and see precisely where muscles connect to the bone to allow flexion, extension, and other movements. To assist with study, instructors can make visual holographic labels to identify organs. By making the labels go blank, students can quiz themselves or take a formal assessment to ensure they are meeting learning objectives.

Surgical faculty can wear the device while performing procedures so that students can "see" the perspective of the expert, rather than through the lens of a mounted camera. Vice versa, a student performing a procedure or exercise can get the advice of experts from around the world in real time, with real interaction. Students are able to see how all of the body systems function separately and together.

Holographic anatomy is an opportunity to revolutionize how health professionals around the globe will learn about medicine and come to understand the human body. It creates a pathway for learners, even from very resource-limited countries, to be able to access world-class knowledge and training anytime and anywhere with a Wi-Fi connection.

CWRU and its partners are now working with faculty on approved content creation and are focusing on evidence for efficacy through outcomes-focused pilots. Early results are very positive and show that students learn at least as well, in less time, than they did in the cadaver lab. Certainly, there are other very important issues for medical students to learn while working in the lab, but the "living anatomy" lessons are proving fruitful.

The IC team is developing HoloLens applications for disciplines such as genetics, chemistry, art, dance, engineering, and paleontology. Students will have opportunities to see and experience conditions and techniques never before available within their own classrooms.

Anatomy represents just one example of the extraordinary potential for learning through MR. It is time to start to think of new ways to utilize this technology, and other such future developments, for teaching and learning. Classroom designs will change, perhaps even becoming empty rooms for holograms to fill. Instructional designers will have a brand-new palette to work with for presenting new and fascinating student experiences. Online learning will have a meaning entirely different from the physical distance it often represents today. Demonstrative learning is a reality.

This work has been featured in numerous publications and has won recognition through several award programs.¹ However, the biggest reward for us will be seeing our students learn better and faster because of the cutting-edge work being done.

Note

1. See "Interactive Commons Team Displays HoloLens Exhibit at World Economic Forum's 47th Annual Meeting," The Daily (Case Western Reserve University), January 19, 2017; Nina Strochlic, "Scientists Are Turning Your Body into Holograms," National Geographic, June 2017; Leigh Kamping-Carder, "How Holograms Are Helping Medical Training," Wall Street Journal, July 5, 2018; Ross Kohan, "Here's How Augmented Reality Is Being Used in Medicine," [http://fortune.com/video/2017/05/03/heres-how-augmented-reality-is-being-used-in-medicine/] Fortune, May 3, 2017; "Virtual Reality Check," CBS News, Sunday Morning, January 8, 2017; Katie Couric, "Cleveland Rocks: No 'Mistake' about It," Yahoo News Video, June 7, 2017; Kathryn Jeffords, "Virtual and Augmented Reality: Changing the Game in Healthcare," SMASH: Science Media Awards & Summit in the Hub (website), June 29, 2016.

---

Sue B. Workman is Vice President of University Technology/CIO at Case Western Reserve University.