The Columbia University Mailman School of Public Health is leveraging the remote teaching versions of its core curriculum courses to develop a high-quality, fully online master of public health (MPH) program.
The Digital Learning Studio at the Columbia University Mailman School of Public Health took the opportunity provided by the emergency switch to remote instruction during the COVID-19 pandemic to identify possible courses for new online programs. The Mailman School of Public Health systematically reviewed the remote teaching version of its core curriculum courses and used that process as the starting point for the development of a high-quality, fully online master of public health (MPH) program.
The emergency remote teaching experience drastically changed the entry point for converting core courses to online instruction. We in the Digital Learning Studio were able to begin the cycle of online course design and development with faculty who had a semester of teaching the course online under their belts. We also had data about student behaviors, satisfaction, and performance in the online versions of the courses and a bevy of multimedia assets that had been created for these courses. Our challenge was figuring out how to wrangle it all. We needed to complete a thorough inventory and review of the data and resources so we could make the most of them.
We developed a multipronged approach that included completing an online course-evaluation rubric, mapping the curriculum, applying a taxonomy, calculating course workloads, and reviewing multimedia assets. We then cross-referenced these data with learning analytics and student evaluations.
Context
Just before the shift to emergency remote teaching, the Mailman School of Public Health had proposed an advanced online MPH program that would allow us to accomplish three things:
- Address gaps in meeting the needs of current on-campus students
- Reach new students
- Develop capacity to diversify educational programs in the School of Public Health
The MPH program includes an integrated core curriculum focusing on foundational knowledge in the field of public health. All MPH students begin the program with a required semester-long introduction to core knowledge in public health, including the historical, ethical, biological, social, and environmental aspects; health systems; and research methodological foundations of population health. The fifteen-credit core curriculum is concentrated into one semester in a strategically integrated, six-course, eighteen-concentration (educational unit) experience.
In practice, over eighteen teaching faculty are needed to accommodate eighteen concentrations, multiple cohorts, and an average of four hundred new MPH students each fall.
Opportunity
All classes offered by the Mailman School of Public Health moved online in mid-March 2020 to foster social distancing, and the majority of courses remain online to this day. "Even in the middle of a pandemic—especially in the middle of a pandemic—public health education continues. The value of public health science and training has never been more evident," Linda P. Fried, dean of the Mailman School of Public Health, said at the time. "We are committed to delivering the same very high quality of education, whether online or in-person, education that is community-based and engaged. I'm so proud of everyone who made this transition happen under these difficult circumstances, and our success in this is a statement about our mission of social solidarity. It is inspiring to see all the ways we can adapt and carry on as a community in the face of a public health emergency."Footnote1
The shift to remote learning was a significant undertaking:
The rapid move to virtual classrooms took a massive, coordinated, behind-the-scenes effort. [Faculty and] teaching assistants mastered the finer points of using Zoom as a classroom, including the use of breakout rooms for small group discussions. The Office of Education took calls, fielded emails and real-time chat messages, and posted an online repository of tool tips and resources for online pedagogy. Human resources and IT staff provided crucial remote support to ensure a smooth learning experience.Footnote2
From the beginning of this effort, the Digital Learning Studio staff channeled these efforts into capacity-building for our longer-term online learning goals. We communicated with faculty early on about the differences between emergency remote teaching and well-designed and developed online learning experiences and worked with them to build the best possible courses. When the rush to create and deliver the remote-teaching versions of the core curriculum courses was over in fall 2020, we immediately turned to the task of thoroughly reviewing what had been accomplished to ready the core curriculum courses for a long-term, fully online MPH program.
Purpose and Goals
To customize this program for working professionals who choose to learn online, we set seven priority goals:
- Manageable workload
- Regular access to faculty
- Attention to public health strategic skills
- Integrated professional context
- Active learning experiences
- Frequent peer interactions
- Authentic assessments
Because online learning programs are new to the school, the Digital Learning Studio staff is also establishing roles and processes as an online learning unit and demonstrating our value to faculty. We set the following goals for the faculty experience:
Design and Develop as a Team
- Collaborate across the full-cycle design and development process of the advanced online MPH courses.
- Reduce faculty workload.
Design for Reusability and Flexibility
- Build efficiencies into the process.
- Create customizable components.
Design for Continuous Improvement
- Embed outcomes tracking into the learning management system (LMS).
- Develop a learning analytics dashboard.
- Systematize the revision and release cycles.
An Iterative Process
Starting with the first course in the core curriculum, with the student and faculty experience in mind, we iteratively established the frameworks, methodologies, and instruments for reviewing the remote-teaching versions of all of our core courses. Our approach was informed by the work of higher education colleagues across the globe:
- University of Virginia's c3Design integrated syllabus framework and Fink's Taxonomy of Significant Learning
- The Course Workload Estimator from Rice University and Wake Forest University
- Dublin Core Metadata Initiative
- OLC Course Evaluation Scorecard
- Equity principles framework informed by the University of Denver Center for Multicultural Excellence toolkit and Patricia Young's Culture-Based Model
Curriculum Mapping
We began by mapping the learning objectives for every course and concentration to the requirements of the accrediting bodies for foundational knowledge areas and degree competencies. We used this mapping schema to provide visual and textual summaries of the types and amount of coursework devoted to each.
Learning Experience Coding
We then coded the learning objectives according to Fink's taxonomy and mapped the coursework activities to those objectives, categorizing each activity as active or passive learning to provide a proportional snapshot of the student learning experience.
Course Workload Calculations
All coursework activities were entered into the Course Workload Estimator to help determine how much time needs to be devoted to each learning objective and to estimate student workload.
Student Evaluations
Student course evaluations provide additional information about how students feel about the number, timing, and relevance of coursework activities and the rigor and inclusion of strategic skills in the curriculum. Cross-referencing student evaluations with data gathered in this review process helped triangulate findings and shape online course conversion priorities.
Multimedia Asset Review
To help create a framework for the audit of multimedia assets for schoolwide online courses and begin converting core curriculum courses for the new online MPH program, we developed and implemented a process to track and organize recorded video and audio, slides, graphics, and interactive media. Recordings of live class sessions were not included in this process. In addition to using the metadata structure recommended by the Dublin Core Metadata Initiative, we included the estimated time to develop each asset, the skills required, alignment with Mayer's principles of multimedia design, reusability and extensibility of the asset, production quality, information about the accessibility of the asset, and whether the asset promotes diversity, equity, and inclusion.
Learning Analytics
We used analytics from the LMS and video content management systems to gain insights into student behaviors with coursework. Clicks on content by students were used as a proxy for the percentage of students engaging with all non-video course materials across cohorts by studio/concentration/unit. Views, duration, and heatmap data provided additional information about student behaviors and engagement with video-based course content.
Outcomes and Next Steps
We are currently working with course faculty leads and concentration teaching faculty to build consensus for course redesign and revisions. It is already clear that faculty find the reviews useful and that the reviews are helping faculty focus their decision-making. For example, one faculty member sent in the following comment:
We were going to add a whole extra synchronous lab to . . . Week 4 content. . . . This might be a good opportunity for something like a watch party/Q&A after—instead of just piling on more new content. THIS IS WHY ANALYTICS ARE AWESOME. Thank you!!!!
The faculty members did follow up and hold an online watch party where students could view the videos for week four synchronously. The faculty members were on hand to answer questions. This example reveals that in addition to the changes that will occur in the online program, faculty insights gleaned from this inventory process are also influencing changes to the in-person MPH core curriculum.
The learning analytics and inventory process has uncovered institutional challenges related to our technology infrastructure and accessing data for our educational technology tools. Gathering learning analytics related to course workload and student engagement in the core curriculum required us to integrate information on student behavior and course activities across all the core classes and cohorts in a single data stream. Though all the relevant datasets are contained within the LMS and video content management systems, the necessary integration had to be performed manually, which was labor-intensive and subject to inconsistencies and errors. For example, the LMS could not pull reports across multiple cohorts of one course. Reports from the course-level displays had to be pulled manually from each LMS course site for four different cohorts and then aggregated into a single dataset.
This challenge prompted us to collaborate with the Academic Technologies Leadership Group (ATLG) at the university and form a subcommittee dedicated to creating an academic reporting environment driven by course-level pedagogical needs. In the future, the subcommittee plans to address challenges related to integrating data from the LMS and other educational technology tools (such as video content management systems, discussion boards, and polls) and monitoring student performance across semesters. Additionally, the subcommittee will develop ways to overcome barriers to university-wide integration of workflows that arise from differences in IT infrastructure across schools and units.
Conclusion
With a focus on student and faculty experiences, the Digital Learning Studio staff is making evidence-based recommendations and suggesting a framework for decision-making to move our work forward. Paying careful attention to student behavior throughout the process has allowed us to improve the student learning experience at the Mailman School of Public Health. Faculty have told us that different components of this systematic approach gain more or less traction in their decision-making depending on their personal pedagogical interests and goals. We have also learned from faculty that what they may pay attention to at first does not always predict what they will show the most interest in as the course-development cycle continues. The review model we developed for the online MPH program is being used for targeted improvements to the current blended and in-person core curriculum courses and can serve as a template for the review and development of other online education programs at Columbia University and at other institutions.
Notes
- "Public Health Learning Continues Online," Public Health Now, Columbia University, Mailman School of Public Health (website), April 15, 2020. Jump back to footnote 1 in the text.
- Ibid. Jump back to footnote 2 in the text.
L. Roxanne Russell is the Director of Online Education at Columbia University.
Lauren Kane Sample is an Instructional Designer at Columbia University.
Shwetha Bhaskar is an Online Instructional Designer at Columbia University.
Peter Lewis is a Digital Media Developer at Columbia University.
© 2022 L. Roxanne Russell, Lauren Kane Sample, Shwetha Bhaskar, and Peter Lewis. The text of this work is licensed under a Creative Commons BY 4.0 International License.