Harnessing the Power of Technology, Openness, and Collaboration

As Lorena steps into her Biology 101 classroom for the first time, she feels a combination of excitement and uneasiness. As the first person in her extended family to attend college, she feels great pressure to succeed. Over the first few weeks of the course Lorena, who is diligent about attending every lecture, struggles with her homework, which includes a more complex vocabulary than she had expected. The decision to buy an old, cheaper edition of the textbook is turning out to be a major problem, as some key chapters and question sets are not the same as in the edition being used in the course.

After performing poorly on the first two quizzes, she wonders if she should drop the course, but since her instructor has not seemed to notice a problem with her work, she decides to push forward. Although disappointed, she is not surprised when she gets a C- on the exam. She ultimately drops the course so that she can return the textbook and get a small portion of her tuition dollars returned.

The following week Professor Halyard, seeing the notice from the registrar about Lorena’s withdrawal, wishes he had connected with her earlier in the semester. With a long sigh he accepts that with the time required to develop each lesson and the size of the course, it is simply impractical for him to be aware of every student’s performance and stay in contact with them.

The Case for a New Outcome

Although the details can vary from institution to institution, the scenario described above is all too common today within our colleges and universities:¹

• Only 28 percent of students pursuing certificates or associate degrees from two-year institutions will complete their programs within three years.
• Of students starting a bachelor’s degree program, 36 percent will complete it within four years.
• For this same population, 58 percent will complete a bachelor’s degree within six years.

As we peer over the five-year education horizon, we see hope in two projects using open educational resources (OER), open academic analytics, and open-source software to address these systemic challenges to college success and completion.

The Kaleidoscope Project, led by Cerritos College, is developing OER-based designs for general education courses across eight partner colleges that serve predominantly at-risk students. The use of OER eliminates textbook costs as an obstacle to student success; it also allows the faculty teams to measure learning results for each student and adapt materials and designs based on the assessment data.

The Open Academic Analytics Initiative (OAAI), led by Marist College, is developing an open-source ecosystem for academic analytics that will include an open API and predictive model for the Sakai Collaboration and Learning Environment (Sakai). OAAI will facilitate the use of a range of open and proprietary business intelligence tools for early detection of risk and early intervention to improve student success.

Separately, these two projects represent important steps forward for the relatively young fields of academic analytics and adaptive OER. Combined, they could facilitate a cost-effective systemic approach for addressing the common challenge of college completion, leading to benefits beyond those originally envisioned by either project.

Unlocking the Gates to Success through OER and Academic Analytics

Both projects build on technology-driven innovations in OER and academic or learner analytics that have emerged in this decade. When combined, they provide a potential key to unlock the gates to the ivory tower of higher education for all students.

Innovation #1: Open Educational Resources

OER may be defined as teaching, learning and research resources that reside in the public domain or have been released under an intellectual property license that permits their free use or repurposing by others.² This “openness” allows educators to reuse, revise, remix, and redistribute academic resources.³ As academic research builds upon prior work to expand the creation of knowledge, OER allow the academic community to build on successful prior work to improve the dissemination of knowledge.

How can OER impact student results?

• Reduced cost. Early anecdotes show benefit from eliminating textbook costs as a barrier to student success among low-income students, particularly in fully online courses. When Cerritos College implemented an open business management textbook from Flat World Knowledge that students could freely access online, completion rates in the traditional face-to-face sections improved by one percent, but completion rates in the fully online sections improved 17 percent.
• Increased flexibility. Instructors can edit textual content that is published under a Creative Commons license to better match the learning outcomes for the course. Materials can be remixed, allowing OER from different sources to come together in support of learning needs.
• Enhanced value. Instructors can more easily incorporate supplemental materials, as additional resources will not create new purchases and costs for students.

With broad investment from higher education institutions and foundations, the breadth, quality, and use of OER is extensive, creating tremendous opportunity among those willing to adopt, refine, and focus OER to improve student success.

Innovation #2: Academic or Learner Analytics

Academic analytics combine select institutional data, statistical analysis, and predictive modeling to create intelligence on which one can act as a means to improve academic success. This holds potential to provide new technological tools for improving course and degree completion.

How can academic analytics impact student results?

• Predict real-time risk to student success. Based on the work of Dr. John Campbell,⁴ Purdue University uses data collected by instructional tools (such as the course management system) to determine in real time which students might be at risk to not complete their course.
• Allow early interventions that shift the students’ trajectory. Once identified, these students can receive “interventions” via notifications sent by their instructor, which guide them to appropriate academic support resources such as online practice exams or tutoring assistance, along with encouragement to use them.

Purdue University has seen significant benefit from this analytics-based early warning system. For example, in a gateway Biology course with 300 students, 12 percent more students earned B and C course grades in sections using the early warning system versus control sections.⁵

The Virtues of Technology, Openness, and Collaboration

Each of the projects individually builds on and advances important work in support of college completion and student success. The two initiatives are collaborating to mature these trends as a way to tackle the growing national college completion crisis.

Kaleidoscope: Collaborative Open Courses

The Kaleidoscope Project was funded by a Wave I Next Generation Learning Challenges (NGLC) grant in May 2011. The ambitious plan required faculty teams from eight colleges across the United States to create common course designs using, wherever available, existing high-quality OER that could be implemented in courses less than four months later.

The project goal for the fall 2011 term was to eliminate textbook costs as an obstacle to success for low-income students. The courses are currently underway, and the Kaleidoscope course designs have reduced costs of the required texts by an average of $60.30 per student per course. This results in a total savings to students of over $60,000 during the fall 2011 term. Upon completion of the term, the project will report whether the elimination of the text costs allowed greater access to the materials, improved completion, and improved student compared to control data from prior terms.

The fall 2011 term also lays the groundwork to measure and influence learning achieved by at-risk students. The Kaleidoscope Project Steering Committee members collaboratively developed a model for the course designs that achieves three key requirements:

• Address, without modification, the student learning outcomes defined and approved through curriculum review processes of each of the eight colleges
• Allow faculty the academic and creative freedom to create unique and diverse learning activities for students
• Create a common assessment approach to measure student learning across the colleges and over time

Based on these requirements, the cross-institutional faculty teams agreed to key areas of alignment within each course team:

• Use a common summative assessment for each student learning outcome
• Use at least one common midstream assessment for each outcome
• Identify OER that support the learning outcomes
• Consider the course design in light of the AAC&U VALUE rubrics as a tool to assess deeper learning

The use of a common assessment and OER provides the faculty teams with an unusual ability to understand and improve student learning. Where students are not achieving the learning outcomes, the faculty teams can revise the OER, evaluate supplementary OER, or remove specific content elements and replace them with content that better supports student learning.

The Kaleidoscope Project has not found a systematic and effective approach to analyze student activity data captured in the course management system to inform the evaluation of the course designs and student learning. While intellectually this shows promise, practically our attempts to use existing tools to this end have not been fruitful.

Beyond the experience with OER, the faculty teams have appreciated the unusual opportunity to collaborate deeply in support of student learning. The diversity of experience and approach among the faculty supports creative new learning activities and provides a more effective evaluation of OER for potential inclusion in the course design. While the open and technological characteristics of OER are a core enabling component of the project, the personal collaboration of the faculty teams shows the greatest promise in addressing the learning needs of at-risk students.

Together, the opportunity for sharing expertise and experiences, the access to a broad range of OER learning assets, and the grounding in data from a common assessment of student learning show promise in arming faculty to best address the complex challenges of at-risk learners.

OAAI: Collaborating Toward Improved Interventions

The Open Academic Analytics Initiative, also funded by a Wave I NGLC grant, has three key project deliverables:

• Create an open predictive model that identifies, based on student demographic, aptitude (such as SAT/ACT scores), and course management system (CMS) data, students who are at risk of not successfully completing their course
• Create an open API for Sakai that improves institutions’ ability to draw “student effort data” (such as activity in Sakai, grades, etc.) from Sakai that will feed the predictive model
• Design, deploy, and measure the effectiveness of a range of student intervention strategies aimed at improving the academic success of students who have been identified as being at risk of not completing their course

Releasing an “open predictive model” through an open-licensing structure using the Predictive Model Markup Language (PMML) will facilitate enhancement of the model by others over time, resulting in an increase of its predictive power. An open predictive model could also lead to the development of a predictive model library from which institutions can select. For example, the library might contain a predictive model designed specifically for community colleges with a primarily first-generation college student population or one designed for use at public universities with large class sizes.

OAAI will soon release an open API to automate the extraction of student effort data, such as event log and gradebook data, required by the predictive model. Initially, the API will be fairly basic and provide just the data needed by the initial predictive model for accurate analysis. The open nature of the API will facilitate further development over time by the Sakai open-source community, leading to a more complex, configurable version. This approach will allow institutions to configure the API to deliver the data required by a predictive model customized to their academic context. For example, predictive models designed for use in fully online courses might need different usage data than those designed for face-to-face classes.

Predictive models and APIs will not affect course completion and retention rates without being combined with powerful intervention strategies aimed at helping at-risk students succeed. To address this, OAAI has developed a concept called an Online Academic Support Environment (OASE) that leverages Sakai Project Sites to provide students with a support community and resources aimed at aiding academic success. Resources include:

• OER content to build study skills and time management
• OER that supports remediation efforts
• A professional academic support specialist
• A student mentor

OAAI has released an OASE Design Framework along with a prototype to facilitate development of localized versions at each of its partner institutions that will be piloting the OASE and analytics in spring 2012. Both will be shared at the 2012 EDUCAUSE Learning Initiative (ELI) conference.

A research project has also been designed to measure the impact of participation in OASE on course completion rates of students identified as being at risk. In addition to examining the effects on completion, the research will also explore the impact that involvement in an online support community has on student engagement levels as measured by the National Survey of Student Engagement (NSSE) instrument. These results will be of particular interest given significant evidence from the past 20 years of research⁶ that high levels of student engagement or involvement with their institution is empirically linked to higher rates of student retention

The academic analytics ecosystem that is emerging from OAAI will provide institutions with a suite of powerful open technologies that when combined with new and innovative intervention strategies hold great potential to positively impact course completion, content mastery and semester-to-semester persistence. Most significantly though, when this academic analytics ecosystem is combined with Kaleidoscope’s OER-based General Education curriculum, common assessment strategies and faculty-based learning communities it will produce a comprehensive system for improving academic success.

The Power of a Systemic Approach to Academic Success

While each project promises benefit, the greater potential to influence student completion and success results from a systemic approach that combines the two. This systemic approach can bring together a more complete data set that includes activity data, assessment data, student engagement, and achievement of deeper learning outcomes. This rich data will inform improved interventions and suggest adaptations to course designs and open resources that better support learning. Where faculty teams from across institutions collaborate in reviewing and acting on data, the pace of learning and improvement will speed dramatically.

We have identified two scenarios that could result from the collective benefit of OER and academic analytics. Each would create a different scenario for our biology student, Lorena, and others like her.

Scenario 1: Analytics Driven Subject-Specific Remediation

Between her part-time job and her course work, Lorena has been struggling to find time over the past two weeks to sign into the learning management system and read her course materials. When Lorena completes her first quiz with a poor score, a note included with the score suggests she may not be reading the material or spending sufficient time studying outside of class. The instructor invites her to speak with him, but she feels embarrassed and vows to study harder for the next quiz.

After completing the second quiz, Lorena knows that she didn’t do well and expects another poor score, but she doesn’t ever see that score. When she tries to see her quiz score, she is encouraged to complete a self-assessment with the promise that she may retake the quiz once she has completed it. She scores herself low on her ability to understand several key concepts within the text and finds that she lacks the mathematical knowledge needed to solve some of the algebraic equations presented to her.

The self-assessment immediately provides Lorena with feedback, suggesting that she watch several short videos. The first video is a graphical simulation of mitosis with narration that explains what she is viewing. The visual explanation seems to help Lorena’s understanding of the concept and allows her to read her text with greater understanding. She is also provided with access to web-based software that provides scaffolding exercises aimed at helping her master solving algebraic equations. With this refresher, Lorena feels much more confident.

The following day Lorena retakes the second quiz, scoring significantly better than she did on the first quiz.

Scenario 2: Supporting Faculty in Supporting Students

Professor Halyard pulls up his dashboard in the Open Analytics Support Environment to see how students have performed on the first quiz. While the weighting isn’t heavy, this quiz provides an early indicator of those students who are struggling to master the basic vocabulary and concepts. Several students have performed poorly, Lorena among them.

Professor Halyard quickly reviews the standard message that the system will send to these students — a wake-up call of sorts — and looks at the second quiz. Should performance remain low on the second quiz, the results will be more problematic.

He connects to the collaboration space for his faculty learning group. The faculty members in the group have each focused on one learning outcome, identifying supplementary open resources that address the most common failings of students over the past three terms. Professor Halyard quickly selects those that have shown greatest impact on completion and success for the topics in the second quiz and moves them into the resources links that will be part of the intervention message to students having poor scores on that quiz.

He notes a comment by a peer at another college who is part of his collaboration team. “The course materials are very text based,” she writes. “When students are struggling with comprehension of technical texts, use of supplementary video seems to help them understand the concepts well enough that they can return to the text and read with greater comprehension.”

He pauses. The predictive modeling and review of analytics from previous courses are allowing him to more quickly pinpoint troubled learners. The breadth and quality of resources that his faculty collaboration team has amassed provide options for most learning needs related to his course. Why wait, he wonders, for performance to drop before we begin to target specific learning activities and resources to match the style, needs, and challenges of each learner? He turns back to the Open Analytics Support Environment dashboard with a new vision for shaping successful learning for his students.

Conclusion

Predicting the future is risky business, especially when technology and people are involved. These specific scenarios may never see the light of reality, but today’s trends toward open learning, collaboration technologies, and open academic analytics could clearly form the foundation on which to build such a future. In this future a student’s use of learning technologies to access course content, interact with peers and instructors, and engage in assessment activities produces “effort” data that feeds powerful analytics able to identify not only which students are struggling academically but also the specific subject matter or skills challenging them.

Armed with such information, customized interventions could be deployed that leverage the power of faculty-based learning communities who organize around specific academic challenges and work to adapt OER to help students overcome them.

Even if some of these predictions become realities, we will see a future in which students no longer need to scale the walls of the Ivory Tower on their own but are provided with the academic scaffolds needed to ensure their success.

1. U.S. Department of Education, National Center for Education Statistics, Digest of Education Statistics, 2001–02 to 2007–08 Integrated Postsecondary Education Data System, Fall 2001, and Spring 2002 through Spring 2008, prepared June 2009.
2. Daniel E. Atkins, John Seely Brown, and Allen L. Hammond, “A Review of the Open Educational Resources (OER) Movement: Achievements, Challenges, and New Opportunities,” Menlo Park, CA: The William and Flora Hewlett Foundation, (February 2007), p. 4.
3. John Hilton III, David Wiley, Jared Stein, and Aaron Johnson, “The Four R's of Openness and ALMS Analysis: Frameworks for Open Educational Resources,” Open Learning: The Journal of Open and Distance Learning, Vol. 25, No. 1 (February 2010), pp. 37–44.
4. John P. Campbell, Utilizing Student Data within the Course Management System to Determine Undergraduate Student Academic Success: An Exploratory Study, Doctoral dissertation (UMI No. 3287222), Purdue University, 2007. 
5. Kimberly E. Arnold, “Signals: Applying Academic Analytics,” EDUCAUSE Quarterly, Vol. 33, No. 1 (2010).
6. George D. Kuh, Jillian Kinzie, Jennifer A. Buckley, Brian K. Bridges, and John C. Hayek, “What Matters to Student Success: A Review of the Literature,” National Postsecondary Educational Cooperative (July 2006).

© 2011 Josh Baron and Kimberlee Thanos. The text of this EQ article is licensed under the Creative Commons Attribution 3.0 license.