An Adaptive Learning Partnership

At its optimum, adaptive learning personalizes instruction, using digital courseware to customize content, assessment procedures, and student preferences for acquiring information. It has been called "the next big thing" because rapidly developing technologies maximize the potential for increasing achievement and eliminating course bottlenecks in higher education.¹

As with so many technological innovations, people sometimes assume that AL is a breakthrough as well as new development. However, adaptive learning is anything but new — the idea has been around for well over five decades, and, if the emerging research is any indication, it will endure for several more. Since John Carroll's formulation of the adaptive process,² many individuals and organizations have recognized that if educational time is held constant, student learning will be the variable. However, altering the paradigm so that learning becomes constant makes time the variable. This seemingly simple concept creates considerable complexity for our educational system in its current configuration, posing issues for such things as course length and financial aid. As Jay Forester pointed out, when an intervention is introduced into a complex situation (such as a university), it is virtually impossible to predict how it will ripple through the system. Further, outcomes can be counterintuitive and with side effects, both positive and negative, that will alter prior expectations.³ We argue here that this complexity gives rise to the need to examine adaptive learning from multiple perspectives, involving partners with differing assumptions, perspectives, capabilities, and challenges. These varying viewpoints add context to help understand how emerging technologies can best address the complexities we face.

The Evolution of an Adaptive Partnership

Often, educational environments that develop around an innovation like adaptive learning involve emerging technologies. The obvious players in such a culture are the universities that buy the platforms and the vendors who market them. Historically, in this configuration all parties involved perceive their relationship from common, but also varying, perspectives. Universities develop strategies for initiatives they consider responsive and transformative. However, most institutions have diverse value propositions depending on their research agendas and the communities they serve. On the other hand, although vendors may work closely with higher education professionals who can advise them on the development of their products, their perspectives about how the educational environment functions can be mediated by their business propositions.

Early research⁴ indicates that adaptive learning has the potential to serve many instructional roles including mentoring, assessment, feedback, course granularity, predictive analytics, and several more learning functions.⁵ Universities confront these initiatives differently depending on their approaches to increasing retention, improving graduation rates, and maintaining quality learning. Vendors have to evaluate their platforms by delivering instruction in a cost/benefit environment. Achieving these capabilities involves compromise by universities and vendors forming collaborative partnerships — an essential bond that should become the new normal in higher education.

Two University Partners

Early adopters of adaptive technology, the University of Central Florida (UCF) and Colorado Technical University (CTU) began discussing their approaches at scientific meetings, describing their instructional processes to adaptive learning as well as their implementation strategies. Both UCF and CTU's digital learning initiatives adopted Realizeit's platform as their enterprise solution. Realizeit supports sharing experiences and data that provide insight into successful adaptive learning practices. Both universities experienced positive results with the technology, but at considerably different scales.

After institutional visits, teleconferences, cooperative writing, and collaborative conference presentations, it became clear that a research partnership between UCF and CTU would be beneficial. A fundamental component for successful cross-institutional collaboration was an openness to each other's work in considerably different student populations, faculty composition, and structure. UCF is a large public university (66,000 students) in the pilot phase of adoption, while CTU is a private, for-profit institution (24,000 students) further along with scaling adaptive learning. The initiative at UCF is research intensive, and implementation centric at CTU.

Implementing Adaptive Learning

CTU began developing and delivering adaptive learning courses in 2012 when university leadership made the commitment to have adaptive teaching and learning strategies as part of its long-range academic programming plan. The university implemented the Realizeit Intellipath platform. Initial pilots included math and English, with three courses offered to approximately 100 online students. Now, CTU students commonly experience adaptive learning in multiple ways, including online courses and as part of blended courses, where it assesses a student's strengths and weaknesses and then modifies how the course material is delivered. This way the student spends less time reviewing known material and more time in the areas where support is needed.

UCF launched its adaptive courseware initiative in fall 2014. After evaluating both open and closed (off-the shelf) platforms, faculty preferred to retain control of content and chose the Realizeit adaptive platform, which became the campus enterprise choice. At UCF, adaptive learning undergraduate and graduate courses have been taught in fully online, blended, face-to-face, and lecture-capture modalities. Courses include Psychology, Intermediate Algebra, College Algebra, Pathophysiology, Statistics for Educational Data, Professional Administrative Writing, Computer and Network Security, Local Area Network Technology, and Applied Systems Analysis. UCF research has found that adaptive learning can improve student retention, increase course outcomes, and provide more precise measures of learning.⁶ Student reactions to adaptive learning have been positive, especially its ability to provide more personalized instruction that can allow students to remediate or learn course material more efficiently.⁷

Table 1 compares adaptive learning initiatives at both universities.

Table 1. Adaptive learning at two scales*

	University of Central Florida	Colorado Technical University
Started with adaptive learning	Fall 2014	Fall 2012
Number of adaptive courses	22 (54 instances)	199 (2,870 instances)
Typical course length	12 (summer)/15 (fall or spring) weeks	5.5 weeks
Number of students	4,025	99,611
Number of enrollments in courses	4,298	629,926
Enrollments per student	1.1	6.3

*Data provided by Realizeit; correct as of October 31, 2017.

Partnering with the Vendor

Vendors developing adaptive learning platforms can spend years in the research and development phase. During this time, they configure their products based on a view of how their design and technology can empower the various instructional roles within an institution. However, one of the challenges they face is the range of university contexts. This complexity means that vendors cannot treat a genuinely adaptive learning platform as a simple plug-and-play device. Each implementation must be responsive to the requirements of the courses, instructors, academic standards, and goals of individual institutions.

The most successful integration occurs when an open, collaborative relationship thrives between the vendor and its university partners. While any vendor will have elements they consider proprietary intellectual property, there should be transparency in core areas. These include how the platform makes decisions, how it provides feedback and guidance to learners, how it influences the direction of learning, how it supports the student, and which data points are available. All data points generated while working in Realizeit are owned by the participating institution and made available to it for measuring the impact of its adaptive initiative. This information has been the primary source of insights that have influenced the platform's evolution.

Additionally, Realizeit has witnessed the natural formation of a community of institutions working together, sharing their experiences, and providing assistance and advice to each other. This practice has been a key contributor to progress. Experts in the field sharing what they have learned and experienced with the common goal of improving student learning is the common denominator for improving student outcomes. The case study in this article describes a successful partnership among Realizeit, UCF, and CTU, three organizations that bring varying perspectives to the table. As a vendor, Realizeit has benefited from this collaboration, finding new insights into how students perceive and use the platform, and how it impacts their learning outcomes. The same process has improved the universities' adaptive learning programs.

This level of cooperation provides both challenges and benefits to vendors and universities. A collaborative partnership of this nature requires time to build and a commitment of resources at various levels to support the many different roles within the institution, from instructional designers and instructors to researchers and administrators. This openness represents some risk for the vendor in that all results, either positive or negative, are freely available. However, this makes all parties in the collaboration fully aware of the issues involved in forming a common base on which to make incremental improvements through research and practice. The relationship is complex but productive because universities can do things that vendors cannot, and vendors can do things that universities cannot, allowing their interactions to produce outcomes that exceed the sum of individual parts.

Student Adaptive Behavior

Realizeit researchers conducted a study examining students' behaviors as they engaged with the adaptive learning platform. The primary objective was to better understand how they managed their learning in an environment characterized by freedom, flexibility, and self-determination. In more traditional settings, students' progress through the learning cycle at an approximately uniform pace and usually can't accelerate; conversely, they might fall behind with few options to compensate. The agency of adaptive learning enables a variety of effective behaviors that would not have been possible previously. Examining these behaviors and how they might inform us about students showed promise for increasing the probability of success.

Examining progression, we identified behavior patterns indexed by the proportion of concepts completed in courses over time. Common prototypes emerged across different content domains and university settings (a typical CTU course lasts around five weeks compared to 15 weeks at UCF, except for summer 12-week semesters). However, depending on how the course was structured and how instructors engaged their students, not all prototypes appeared in every course. To assist in understanding these categories, we used animal-based metaphors — a technique described by George Lakoff and Mark Johnson and Katrina Meyer.⁸

The Tortoise and Frog represent the two most common student behavior styles in higher education. They make systematic progress throughout the course — a behavior most likely encouraged by instructors in most disciplines. On the other hand, the Hare and Kangaroo prototypes emerge as a result of self-direction and self-pacing. These students accomplish the majority of their progress in altered timeframes. Since all prototypes tended to complete the course, the primary difference was the manner in which they engaged in the course. Figure 1 shows their different approaches graphically.

Many questions about these prototypes remain unanswered:

1. Are any of them harmful to a student's success?
2. Should any prototype be encouraged or discouraged?
3. Do they have an impact on a learner's retention of knowledge?
4. What motivation underlies each behavior?
5. What are the factors, both internal and external to the student, that allow them to follow a prototypical path?

Interestingly, the Kangaroo prototype creates a potential issue for predictive analytic models. The student remains inactive for large portions of the course, creating the potential for an inaccurate "non-success" forecast. The consequence of such an inaccurate prediction would adversely affect the learner in a predictive analytics "early alert" framework.

While the prototypes describe students who completed the course, some do not finish. When we examine their trajectories, they appear as failed attempts at one of the four prototype behaviors. Rather than complete, at some point they flatline and never come back to life, metaphorically. A pervasive question is why this occurred and what could have been done to prevent it? Can we establish a warning signal that recognizes this flatlining before it is too late? Clearly, much more work is needed to better understand this new learning dynamic. The answers to these questions will comprise an important future component of the ongoing collaboration between UCF, CTU, and Realizeit.

Improving Outcomes for Students

One insight that has become evident from the work of UCF and CTU is that any adopter of adaptive learning must persist in the effort. Adaptive learning is not an instant solution. Its strength lies in the feedback cycle that it provides students, instructors, and instructional designers, allowing them to formulate outcomes through an iterative improvement cycle. To highlight the possibilities, we examined how improvements in both the courses and the adaptive platform have led to an increased attainment level of students taking College Algebra at UCF. For example, the instructor has used what she learned after each course to implement changes in the materials and structure, which produced measurable and significant improvements in student outcomes.

Figure 2 shows the distribution of the proportion of concepts covered by students for two semesters of College Algebra. Spring 2015 was the first time that adaptive learning was used in this course, and fall 2016 is the most recent semester available. The effect of the improvement cycle is evident in the shape of these distributions. In both semesters, student concept coverage varies across the range; however, in fall 2016 students cluster more toward the top of that range.

The impact of the instructor's changes becomes more evident by splitting the students into three cohorts and examining what has happened for each.

• Top 25 percent: These students have moved from covering at least 86 percent of concepts to covering 95 percent or more.
• Middle 50 percent: These students moved from covering between 49 percent and 86 percent in spring 2015 to covering between 61 percent and 95 percent in fall 2016. Both the top and the bottom boundaries achieved more.
• Bottom 25 percent: In spring 2015 these students covered less than 49 percent of the course. By fall 2016 some of these lower achieving students covered up to 60 percent of the course.

All cohorts have improved, the best have gotten better, and even the lower achieving students have improved. Adaptive learning has shifted the curve in this course, benefiting all the students. The key is committing to using data from each iteration of the course to learn what worked, what didn't, and where improvements are needed. This same strategy has been used at a much larger scale at CTU.

Student Reactions to Adaptive Learning

We were interested in students' reactions to adaptive learning and how they might differ across the two institutions. UCF developed a survey as part of their pilot evaluation, and CTU adopted the same survey, allowing the universities to compare reactions. While the demographics and campuses vary, it would be beneficial to know if students at both institutions were equally receptive to adaptive learning. At UCF, 300 students were surveyed in General Psychology, an adaptive course delivered in fall 2014 and spring 2015 with 244 respondents (81 percent response rate). CTU sent their survey to all students enrolled in adaptive courses (14,400) and received 1,140 completed responses (10 percent response rate).

The survey gauged student reactions and experiences with their adaptive classes. It solicited details on their interaction with the platform, including ease of use, helpfulness of feedback, and guidance and accuracy of platform assessment metrics. In addition, the survey captured overall student attitudes about using adaptive learning in instruction, including what was most and least positive about this instructional method and how it affected their interaction with and time spent on the course. Demographic questions allowed examination of differences across student cohorts. Overall, students at both institutions felt positive about their experiences. However, some reactions to adaptive learning illustrated significant differences between students at CTU and UCF.⁹

The majority of students at both schools believed that adaptive learning helped them learn better (figure 3), with a slightly higher percentage at CTU agreeing (82 percent) than at UCF (78 percent). Interestingly, CTU also had more students who felt the opposite (12 percent vs. 6 percent) and fewer who were unsure (CTU, 6 percent; UCF, 16 percent). While the differences between CTU and UCF were significant (p=.00), the trend for both schools indicates that students reacted positively to instruction using adaptive learning.

Students at both schools responded positively to the feedback provided by the adaptive learning platform (figure 4), although once again the difference between CTU and UCF students was significant (p=.00). The CTU students tended to either react positively to the feedback (82 percent) or negatively (8 percent), with fewer indicating ambivalence (10 percent) than at UCF (18 percent). Few UCF students reacted negatively to the feedback provided on objectives (6 percent), and the majority were very positive (77 percent).

Students recognized that the platform became personalized to them over time (figure 5). This survey question had slightly more students who disagreed (UCF, 10 percent; CTU, 8 percent) or were ambivalent (UCF, 17 percent; CTU, 14 percent) than other questions. Perhaps this is due to some ambiguity in what constitutes "personalization." However, the majority of the students were still positive and felt the platform was more personalized to their learning than in other courses they had experienced (UCF, 63 percent; CTU, 78 percent).

The response trend continued about whether students felt more engaged in an adaptive learning course (figure 6). Looking at both venues, a similar proportion of students (UCF, 8 percent; CTU, 7 percent) felt less engaged with their adaptive learning course, and 10 percent more of the CTU students felt their engagement increased (85 percent) than at UCF (75 percent). UCF students tended to be more ambivalent (18 percent vs. 9 percent at CTU). However, students were positive and felt more engaged in the adaptive learning course than in a similar non-adaptive course.

Perhaps the best measure of an implemented innovation is whether students would take a similar course in the future, if given a choice. At both CTU and UCF, students' positive experiences with adaptive learning in an instructional context resulted in the majority (UCF, 78 percent; CTU, 86 percent) indicating that they would take another adaptive course (figure 7). The significant differences across the two campuses (p=.01) held with this question. Students at CTU were less ambivalent (8 percent) than at UCF (13 percent) and also less negative about a future course that incorporated adaptive learning (6 percent at CTU and 9 percent at UCF).

An interesting pattern in the survey responses is that CTU students generally indicated less ambivalence than UCF students. We don't know the cause, but it could result from the differences in demographics or that CTU students generally have had more exposure to adaptive learning and therefore are more certain of their position.

Adaptive Leaning Finally Finds Its Place

Over the past few years, higher education has been able to refocus on adaptive learning because emerging technologies have created more effective capabilities. Effective partnerships can emerge across several different providers of adaptive platforms, a development that, in our opinion, would be highly beneficial in avoiding the "which is the best platform?" narrative. More to the point is a thoughtful examination of adaptive learning as an instructional process. For example, a department at UCF decided to investigate two platforms in an A-B comparison, hoping to identify a clear winner. Of course, it should be no surprise that the results were inconclusive. A did some things well and others not so well, and the same was true for B. Of course, the things done well and not so well were not A-B comparable, so no clear winner emerged. Therefore, the department in question has to evaluate its context, objectives, and challenges to arrive at a decision — a process likely to continue for some time. We can't help but believe that if the two platforms had collaborated with the department in a cooperative research initiative, the results might well have been more informative for all parties concerned. We reemphasize that universities can do things vendors cannot, and vendors can do things that universities cannot, adding value to the inquiry process.

One might reasonably ask, "Why does adaptive learning matter, and what are the potential benefits?" First, there is little doubt that widespread educational inequality exists in our country and that students in the lowest economic quartile have less than a 10 percent chance of graduating from college.¹⁰ Given that a bachelor's degree adds approximately one million dollars to one's lifetime earnings, it seems obvious that educational inequality is a primary contributor to growing disparate economic status in the United States. Therefore, increasing the number of degrees for students living in poverty is an objective to which every educator should aspire. Further, if we accept the assumption that the talent pool in our underserved neighborhoods is just as deep as among those young people living in gated communities, we are wasting millions of minds, and we simply we can't afford to do that.¹¹

Scarcity

What does scarcity have to do with adaptive learning? Sendhil Mullainathan and Eldar Sharif provided the context for the answer: They argue that every day, students living in poverty have to cope with scarcity, meaning they need far more supportive mechanisms than they have available to them. They just have no slack in their lives either financially or temporally, so they perform a constant juggling act in order to get through the day, week, or month, burning up most of their cognitive bandwidth reserves along the way. If just one thing goes off the rails, their whole life structure comes tumbling down, with classes becoming collateral damage.¹²

Our experience has shown that adaptive learning's flexibility and go-at-your-own-pace, with modules supported with granular learning nodes and customization, has real potential to alleviate some of these scarcity problems. Pre-assessment can place students at the proper course entry point, and subsequent assessment can check progress, thereby allowing them to accelerate or review skills not yet achieved at the desired competency levels. Systems that incorporate machine learning can recognize students' learning preferences and present information in a form that optimizes learning potential, providing students some necessary learning slack.

Adaptive learning cannot alleviate poverty, but it can help students who live there. We plan future research to investigate this aspect further.

Learning Assessment

Online learning, in general, has challenged the validity of objective student learning assessment protocols such as multiple choice tests that have their security compromised in the Internet world that Luciano Floridi termed the infosphere. He argued that assuming one is either online of offline is no longer valid. In the day of the infosphere, objective tests are easily debriefed, destroying any of the original psychometric properties including but not limited to reliability and validity.¹³ Adaptive learning, because of its potential for continuous assessment, places even more pressure on our current assessment paradigms. Quite simply, adaptive learning forces us to reconsider assessment models that are objective, non-authentic, and non-contextual in favor of those that are more reflective, authentic, and contextual, giving renewed credibility to the authentic assessment literature of the 1990s.¹⁴ Ryan Baker, however, has argued that although the machine learning capabilities of adaptive platforms have made remarkable progress, the assessment procedures have remained largely heuristic, assuming competence if a student responds to two out of three or four out of five questions correctly. Although this might seem like a problem in adaptive learning, it presents us with the opportunity to overhaul our learning assessment procedures to better reflect the knowledge and skills our students will require. This seems, to us, another situation ripe for university-vendor collaboration.

Adaptive Analytics

Another potentially positive outcome from adaptive learning is predictive analytics in real time or very close to it. UCF, working with CTU and Realizeit, has demonstrated the feasibility of adaptive analytics in two areas.¹⁷ First, by tracking average module scores in psychology, a group of students that most likely will not succeed in the course can be identified by the decrease in the mean scores of module two compared to module one. However, the variability in non-successful cohorts is larger than for those who succeed. By deconstructing the non-success groups, we find four prototypes: late momentum loss, early momentum loss, steady decline, and flatline. The last two can be identified almost immediately, while the first two cannot be distinguished from successful students until they reach their critical points. They require constant vigilance. Second, a large number of metrics produced by the Realizeit platform might enable adaptive analytics. Some of these indices have shown themselves equally effective at identifying students who might not succeed. At the moment, the three partners are exploring how close to the course starting point these measures can be and remain effective predictors. Both approaches look promising because they address the problem that chaoticians face: initial starting points have a profound effect on eventual outcomes.¹⁸ Real-time, predictive, adaptive analytics is worth exploring with other vendors as well.

Looking Ahead

As with most potentially good ideas, adaptive learning should be incrementally developed to encompass principles outlined by Johnson:

1. The adjacent possible: What is the reasonably obtainable next step?
2. A slow hunch: A long-term approach
3. A liquid network: A support system for idea interaction and exploration¹⁹

Adaptive learning offers many possibilities, but it might not prove equally effective in all disciplines — for instance, those subject areas with little or no hierarchal structure. This aspect of adaptive learning should be much more carefully examined. Additionally, there is a danger in initial overenthusiasm, where expectations far exceed the capabilities of the innovation. In his introduction to the 50th anniversary edition of Thomas Kuhn's 1970 classic work, The Structure of Scientific Revolutions, Ian Hacking described how the term paradigm shift quickly spun out of control and became so overused that its significance was soon lost.²⁰ We don't want this to happen with adaptive learning.

Although the partnership has strengthened our conviction that adaptive learning offers real potential for educational transformation and improvement, cause-and-effect thinking rivals the expectation of immediate results in its overzealousness. Earlier we argued that universities are complex systems with embedded nonlinear components that make accurate prediction extremely difficult, if not impossible.

Our understanding of adaptive learning will be enhanced by examining multiple perspectives in a careful, systematic process where we interpret findings with a voice that resonates with higher education. As Nate Silver pointed out, data have no voice of their own.²¹ We have to provide that voice.

Notes

1. Charles Dziuban, Patsy Moskal, and Joel Hartman, "Adapting to Learn, Learning to Adapt," Research Bulletin, EDUCAUSE Center for Analysis and Research (ECAR), September 30, 2016.
2. John B. Carroll, "A Model of School Learning," Teachers College Board, Vol. 64 (1963): 723–733.
3. Jay W. Forrester, "System Dynamics and the Lessons of 35 years," in Systems–based Approach to Policymaking, K. B. D. Greene, ed. (Norwall, MA: Kluwer Academic, 1991).
4. Charles Dziuban, Patsy Moskal, Jeffrey Cassisi, and Alexis Fawcett, "Adaptive Learning in Psychology: Wayfinding in the Digital Age," Online Learning, Vol. 20, No. 3 (September 2016): 74–96; Charles Dziuban, Patsy Moskal, Constance Johnson, and Duncan Evans, "Adaptive Learning: A Tale of Two Contexts," Current Issues in Emerging eLearning, Vol. 4, No. 1 (2017), Article 3.
5. Benjamin Samuel Bloom, "The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring," Educational Researcher, Vol. 13, No. 6 (1984): 4–16.
6. Dziuban, Moskal, and Hartman, "Adapting to Learn, Learning to Adapt"; and Dziuban, Moskal, Cassisi, and Fawcett, "Adaptive Learning in Psychology."
7. Dziuban, Moskal, Johnson, and Evans, "Adaptive Learning: A Tale of Two Contexts."
8. George Lakoff and Mark Johnson, Metaphors We Live By (Chicago, IL: University of Chicago Press, 2003); and Katrina A. Meyer, "Common Metaphors and Their Impact on Distance Education: What They Tell Us and What They Hide," Teachers College Record, Vol. 107, No. 8 (2005): 1601–1625.
9. Dziuban, Moskal, Johnson, and Evans, "Adaptive Learning: A Tale of Two Contexts."
10. Jeffrey J. Williams, "College and the New Class Divide," Inside Higher Ed, July 11, 2016.
11. Lizette Alvarez, "One Man's Millions turn a Community in Florida Around," New York Times, May 25, 2016.
12. Sendhil Mullainathan and Eldar Shafir, Scarcity: The New Science Of Having Less And How It Defines Our Lives (New York, NY: Times Books, Henry Holt and Company, 2013).
13. Luciano Floridi, The 4th Revolution: How the Infosphere Is Reshaping Human Reality (Oxford: Oxford University Press, 2014).
14. Wiggings, G. (1993). Assessment: Authenticity, Context, and Validity. The Phi Delta Kappan, 75(3), 210–214.
15. Ryan S. Baker, "Stupid Tutoring Systems, Intelligent Humans," International Journal of Artificial Intelligence in Education, Vol. 26, No. 2 (June 2016): 600–614.
16. Floridi, The 4th Revolution.
17. Dziuban, Moskal, Johnson, and Evans, "Adaptive Learning: A Tale of Two Contexts."
18. Leonard Smith, Chaos: A Very Short Introduction (Oxford: Oxford University Press, March 2007), 35–38.
19. Steven Johnson, Where Good Ideas Come From: The Natural History of Innovation (New York: Riverhead Books, 2010).
20. Thomas S. Kuhn and Ian Hacking, The Structure of Scientific Revolutions: 50th Anniversary Edition/Edition 4 (Chicago: University of Chicago Press, 2012).
21. Nate Silver, The Signal and the Noise. Why So Many Predictions Fail — but Some Don't (Penguin Books, 2012).

---

Charles Dziuban is director of the Research Initiative for Teaching Effectiveness, University of Central Florida.

Patsy Moskal is associate director of the Research Initiative for Teaching Effectiveness, University of Central Florida.

Connie Johnson is chief academic officer and provost at Colorado Technical University.

Colm Howlin is a researcher at Realizeit.