Under the Hood of a Next Generation Digital Learning Environment in Progress

Imagine you have 2,000+ students and need to organize 115 sections of the same course. How do you ensure consistency (of content and assessment) between sections? How do you update course materials on the fly? How do you find and help the students who are struggling? To address all of these questions, put into action a next generation digital learning environment. Here we lay out a conceptual approach to a new NGDLE developed at the University of Notre Dame, describe its implementation, and discuss some outcomes and lessons learned.

Context and Course Model

In the spring of 2015, the University of Notre Dame formed a campus-wide committee to design a new first-year experience (FYE) course that would replace the 30-year-old physical education (PE) requirement. Where the PE requirement consisted of two required physical education courses that included topics intended to facilitate student adjustment to college life, the new course would focus almost entirely on acclimating students to the Notre Dame experience, including the institution's academic and spiritual life, campus resources, cultural diversity, and tools available to build a healthy lifestyle and successful future. A campus-wide initiative, the required FYE course unifies the expertise of various groups around campus in service to the university's mission of teaching, learning, and student success. As such, the FYE course was the perfect setting in which to develop our digital learning environment. From the beginning, it was clear that learner-centered pedagogy, a consistent curriculum, and the innovative use of technology would be vital for the successful implementation of the course.

The course is one credit each semester, meeting once weekly for 50 minutes. With over 100 sections and section instructors, the course requires a common curriculum (course materials, assessment standards, and associated grading rubrics) to ensure consistency of information and parity of student effort and instructor grading. Prior to the discussion-based class meetings, students prepare by completing assigned readings, viewing assigned videos, and/or composing short written responses to assigned prompts. Major assignments, which require reflective and integrative thinking related to course goals, are completed in course-related e-portfolios and shared with course instructors and advisors.

The challenge is that although 85 percent of faculty use a campus learning management system (LMS),¹ a recent Blackboard report found that, out of 70,000 courses across 927 North American institutions, 53 percent of LMS usage was classified as supplemental (content-heavy, low interaction) and 24 percent as complementary (one-way communication via content/announcements/gradebook).² Only 11 percent were characterized as social, 10 percent as evaluative (heavy use of assessment), and 2 percent as holistic (balanced use of all previous). Our FYE course required innovating beyond the supplemental course-level LMS to create a more holistic cohort-wide NGDLE in order to fully support the teaching, learning, and student success missions of the program.The key design goals for our NGDLE were to:

• Create a common platform that could deliver a standard curriculum and achieve parity in all course sections using existing systems and tools and readily available content
• Capture, store, and analyze any generated learner data to support learning assessment, continuous program improvement, and research
• Develop reports and actionable analytics for administrators, advisors, instructors, and students

NGDLE Dimensions in Action

As a guiding framework for the digital architecture of the new FYE course, we followed the five dimensions of an NGDLE from an EDUCAUSE Learning Initiative report³ to close the gaps between current learning management tools and digital learning environments. Initially, we focused our implementation on the "Interoperability and Integration" and the "Analytics, Advising, and Learning Assessment" dimensions, with some attention to "Personalization." We plan to implement the remaining dimensions in future years to complete our NGDLE.

Dimension 1: Interoperability and Integration

The ELI report concludes with the analogy of the "NGDLE as Lego Sets."⁴ We chose to create an open-source–based solution for FYE using Sakai as the main hub and platform to connect various tools ("Legos") instead of paying for a vendor-provided custom solution.

Our level of connection to these tools was not always direct; it ranged from very basic embedding with iFrames or hyperlinking to using more advanced vendor-provided application programmer interfaces (APIs), Learning Tools Interoperability (LTI), and Experience API (xAPI) connections. Figure 1 summarizes the tools used for our FYE course and the general path for system data.

Table 1 details the tools used for these connections between systems and how they were integrated initially to create our overall learning ecosystem for FYE.

Table 1. Initial integration of connectivity tools

System	Level of Integration	Affordances
Learning management system: Sakai 10	Main hub and platform for all tools to plug into	Hub for single sign-on and access to other tools
E-portfolio: Digication	Linked; will use LTI next academic year	Student websites for reflective assignments integrating text, media, course content, and experiences; content shared with advisors and faculty.
Student information system: Banner	API	Account sync
Video-on-demand: Kaltura	LTI	Video player with built-in analytics collection
Document sharing: Google docs	Linked (iFrame embed)	Live and editable documents shared in common with all sections
Rubric and assessment: iRubric	Not used initially, but will pilot LTI next academic year	Provides finer-grained detail to students without increasing instructor time-on-task
Digital badge: Credly	LTI to Digication with LTI to Sakai in progress	Motivate and recognize excellent student work
Data storage and analysis: Apereo OpenLRS	xAPI to Sakai	Create central repository for analytics
Reporting: Elastic Kibana, Excel, Tableau	xAPI Elastic Kibana to Apereo OpenLRS	Live and retrospective course data streams

Dimension 3: Analytics, Advising, and Learning Assessment

Our NGDLE was carefully designed to allow the internal collection of various sources of tool data like grades, click data, and e-portfolio assignment text. Figure 2 shows the range of data provided from three sources within the NGDLE.

The consistent curriculum and assessment standards across the course enabled the uniform collection of data into our learning data warehouse using manual exports, machine learning techniques, and automated methods. Analysis has been done manually through Microsoft Excel, Tableau, and R, and in the past year we have evolved to some limited automated reporting using Kibana. One example of a useful report for the advisors and instructors was course-wide grade distribution, which allowed faculty to benchmark their own grading practices and advisors to contextualize their advisees' scores. Grade distributions also allow program administrators to discern in which aspects of the course students still need help meeting the expectation of mastery.

Dimension 2: Personalization

Despite the standardization of course content and grading rubrics, this NGDLE allowed us to incorporate some elements of student personalization (dimension 2 of the themes). All course materials were available from the first week of each semester, and assignments opened several weeks before due dates, allowing students to move at their own pace. E-portfolio assignments required incorporation of some media, but students had free choice as to the extent of their media production and the type of media they produced. Many students leveraged their ownership of course e-portfolios to update the visual design, and a few added unsolicited custom content.

Impact

Because we retained the LMS as the hub connected to our student information system, which is integrated with all of our other tools, it served as the ideal funnel to capture massive amounts of learning data. Figure 3 shows how our NGDLE comes together to create a data flow that allows for a rich and diverse data ecosystem with a pulse that can create dynamic data visualizations of student learning.

We were able to meet and exceed our key design goal of having a common platform where we could capture all learner data generated, enabling the timely analysis and reporting of data to our stakeholders through both retrospective analytics and real-time reporting of actionable recommendations.

Over the past two years, our NGDLE captured 120,000 graded student assignments, 4.6 million click tracks, and 30 million text-mined words. We generated numerous reports for FYE faculty, advisors, and administration, including visualizations of resources (articles and video), click tracking and heat maps, grade distribution charts by assignment, word frequency tree maps, and word clouds of e-portfolio reflections. Also, we produced lists for advisors identifying at-risk students at mid-semester that included diagnostic contexts for why students were at-risk (e.g. class participation, poor assignments, etc.), allowing advisors to be more productive and focused in their meetings with students. Collected data also allowed direct outreach to those students who had difficulty navigating course technology.

Since its inception, our NGDLE and its products have resulted in several effects for the FYE course or the campus at large:

• Live data collection allows regular pulse-checks on students. Quick data analysis and reporting to faculty, advisors, and administration enables instant teaching and curriculum updates during the semester.
• Success of this learner-centered design provides a widespread model to campus faculty showing how thinking of the LMS as a hub for NGDLE can benefit their students and teaching.
• Widespread exposure to meaningful and actionable learning analytics sparks campus conversations about their use in other courses and settings.
• Anecdotal reports indicate that the choice to have single portfolios shared with faculty and advisors allowed both parties to make deeper connections with their students.
• Sharing student work and examples (enabled by the NGDLE) between semesters and enrollment years contributes to higher-level work by subsequent students.
• The university can also show an evidence-based process for accreditation of what students are learning and producing for courses, which changes the campus conversations from being solely grade-focused and promotes a richer understanding of the student learning experience.

Future Plans

As we plan for the next steps in our emerging NGDLE, we are looking to improve along existing dimensions and branch out to new ones.

• Implementing new LTIs will streamline the student and faculty online experience (Dimension 1: Interoperability and Integration).
• Analytics improvements will permit sharing results earlier in the semester, with an ultimate goal of live updates via a dashboard for students, faculty, and administration (Dimension 3: Analytics, Advising, and Learning Assessment and Dimension 2: Personalization).
• Social learning with e-portfolio assignments or online discussion have been discussed as an evolution of the course (Dimension 4: Collaboration).
• Videos used as course material would ideally be updated with closed-captioning (Dimension 5: Accessibility and Universal Design).
• Strengthening our text-mining procedures (Dimension 3) will allow deeper assessment of student progress toward course goals, which we expect to influence curricular changes and updates to the digital ecosystem (Dimension 1: Interoperability and Integration).

Lessons Learned

Pursuing an NGDLE for a brand-new FYE course that would grant more credit hours (4,000+) than any other single unit on campus provided the perfect large-scale opportunity to unite a dozen people across three units (Center for Teaching and Learning, Academic Technologies, Sakai Group) to spend thousands of hours over months to design, develop, implement, and evaluate five identified dimensions that comprise the environment. A reevaluation of the hope that one vendor will provide a one-stop shop, killer app, or perfect suite of tools to solve all user needs is in play.

Here we use the NDGLE paradigm to rethink our existing course-level learning management system as an integrated platform-level learning ecosystem. The "Lego" approach allowed us to bootstrap a set of applications that enabled transforming use of the LMS from an administration and one-way communication supplement to a more holistic platform with heavy use of assessments. In addition, because we had an open-source LMS, we had a flexible hub with a campus support staff that could quickly, inexpensively, and safely begin to innovate at scale without high vendor cost or long development times.

Lastly, the dimensions provided guideposts for us to continue to move on our journey toward the constantly evolving NGDLE destination. This allowed us to realize that the entire NGDLE does not need to be fully integrated with xAPI or LTI (link/embed can be a start), nor does all data extraction and analysis need to be automatically visualized in real time (manual export and reports also work). Most important is that we have a beta vision in progress with a flexible architecture that can be adapted to the changing needs of higher education.

Acknowledgments

Many thanks to Chris Clark and Kevin Barry from the Kaneb Center for Teaching and Learning; Pat Miller, Laura Gekeler, and Xiaojing Duan from the Sakai LMS Team; Trunojoyo Anggara and Maureen Dawson from the First Year of Studies; Jonathan Finkelstein from Credly; and Joan Watson and Jeffrey Yan from Digication.

Notes

1. Eden Dahlstrom, D. Christopher Brooks, and Jacqueline Bichsel, The Current Ecosystem of Learning Management Systems in Higher Education: Student, Faculty, and IT Perspectives, research report (Louisville, CO: ECAR, September 17, 2014), 3.
2. John Whitmer, Nicolas Nuñez, Timothy Harfield, and Diego Forteza, "Patterns in Blackboard Learn Tool Use: Five Course Design Archetypes," Blackboard, 2016.
3. Malcolm Brown, Joanne Dehoney, and Nancy Millichap, "The Next Generation Digital Learning Environment. A Report on Research," ELI Paper (Louisville, CO: EDUCAUSE Learning Initiative, April 2015).
4. Ibid.

---

G. Alex Ambrose serves as professor of the practice and associate director of ePortfolio Assessment in the Kaneb Center for Teaching and Learning at the University of Notre Dame.

Kevin Abbott is the lead Teaching, Learning, and Technology Consultant in the Learning Platforms group at the University of Notre Dame.

Alison Lanski serves as e-portfolio specialist in the Kaneb Center for Teaching and Learning at the University of Notre Dame.