Helping Courseware Achieve Its Potential in Higher Education

Despite motivation and incentives to succeed academically, many students struggle to balance education with work, family, and other demands, and they lack the confidence and resilience to stick with their educational paths when adversity arises. These "nontraditional" students drive a growing proportion of enrollments even as institutions face new financial and regulatory pressures, making it vitally important to solve the problems hindering their success.

Digital courseware's greatest potential lies in serving these students, asserts Karen Vignare, a higher education executive and founder of KV Consulting. Her experience, including leadership roles at institutions like University of Maryland University College and Michigan State University, lead her to believe that the biggest promise of courseware is providing improved support to students unprepared for college, not just through remediation but also by bringing immediacy to their learning experience:

"There are students that are unprepared and uncertain about how they are doing in their education. While many faculty are overwhelmed and struggle to respond to every student's individual needs, they can use tools like courseware to help students understand their own progress whenever they would like."

Despite its promise, digital courseware has lagged in realizing and scaling its potential in higher education. According to Nielsen Pubtrack, a resource that captures sales of postsecondary course materials through college bookstores nationwide, in 2011 digital course materials comprised only about eight percent of unit sales across a selection of high-enrollment introductory courses, where these nontraditional students are most likely to enroll — and struggle. By 2014, that proportion had grown to about  20 percent of unit sales for the same courses.¹ The growth of digital courseware in higher education has occurred slowly and inconsistently, leading us at Tyton Partners and many others to wonder, "What's the holdup?"

Over the past two years, we've analyzed barriers to adoption and levels of satisfaction with digital courseware. Findings from Tyton Partners' 2014 survey of 2,700 postsecondary faculty and administrators point to a disconnect between the courseware user experience and the control and ease of use that faculty and administrators expect in the classroom. In addition, the survey revealed concern over courseware's efficacy — faculty and administrators do not see a clear reward in terms of improved student outcomes from adopting courseware. With these issues in mind, it's easy to see how courseware has not lived up to its promise.

The Market Is Correcting…and Becoming More Complicated

The courseware market now includes solutions much improved in terms of user experience, even in the time since Tyton's first courseware field analysis in 2014. Faculty demand for course material customization and instructional control has led to the evolution of courseware as a category from a handful of one-size-fits-all "black boxes" of content and algorithms to a broad range of interoperable solutions that allow faculty to pair their choice of content with software that supports their preferred pedagogy. Courseware today increasingly integrates different products, improving the potential to customize based on course or institutional needs. Figure 1 shows courseware delivery models.

At the same time, significant energy has gone to assessing and quantifying courseware's effectiveness in improving learning outcomes. Research organizations like SRI International, a nonprofit research and evaluation firm, have published the results of their large-scale analyses, including 2014 and 2016 reports evaluating the impact of the Bill and Melinda Gates Foundation's investments in digital courseware. On a smaller scale, individual institutions evaluate courseware impacts by setting up studies comparing learning outcomes in courses with courseware to those without courseware. Vendors also actively develop new courseware solutions and release reports on their potential impact.

This looks like progress — the supply of courseware products and information on their impacts inching closer to the demand. While the market appears to be correcting a mismatch in supply and demand, the rapid evolution of courseware and varying, parallel strands of research have consequences: a lot more information to comprehend and little established knowledge to help make sense of it. Institutional leaders feel the pain.

Kara Monroe, vice president of Academic Innovation and Support at Ivy Tech Community College, noted that the challenge is acute around the language used to describe courseware. According to Monroe, the lack of a common definition of courseware or a consistent understanding of product features makes conversations about courseware needs and capabilities inside the institution or with vendors particularly tough. The rapid evolution of products means that just as faculty members become familiar with a product, the features and their descriptions may change, and the ways that those features work in the classroom change, too. This makes comparing products, sharing information, and evaluating the impacts of courseware challenging, especially when evaluations take place over two to three semesters. Courseware adoption can suffer.

The Solution? More (and Better) Information

In response to the growing challenge of navigating a broader and more complex courseware ecosystem, in the fall of 2015 a team at Tyton Partners, along with the Online Learning Consortium and SRI International and in partnership with the Bill and Melinda Gates Foundation,² set out to develop a resource to help institutional decision makers better navigate the courseware market. We had several objectives for this initiative, each targeting a specific challenge identified in the typical courseware discovery, selection, and adoption processes, as shown in table 1.

Table 1. Challenges and objectives in courseware adoption

Challenge	Objective
Identify courseware in a dynamic sea of digital learning solutions	Provide a consistent definition of digital courseware
Effectively communicate courseware needs and capabilities between faculty, IT, academic leadership, and vendors	Establish a common lexicon for courseware and its functionality
Understand how specific courseware product features can support the teaching and learning process	Build transparency into the learning science behind courseware product design
Connect teaching and learning goals with the best tools to help achieve those goals	Provide recommendations for priority product features to help meet goals for courseware use
Access helpful, objective, current information on courseware products	Build a field-owned resource, shared freely and broadly and regularly updated

The output of our work on this initiative over the past 12 months is the Courseware in Context, or CWiC, Framework.

Other organizations have also developed resources and tools to improve the transparency and efficiency of the courseware discovery, selection, and procurement processes. They provide a forum for discussion, shared resources on product features, and feedback and information on courseware pricing and alternatives. They include:

Digital Learning Network

Sidewalk Hero [http://www.gosidewalk.com/]

LearnTrials

The CWiC Framework: A Resource to Navigate Courseware

While many frameworks in the market are designed to support quality instruction and inform technology use in the classroom, the CWiC Framework fills a gap around digital courseware products and implementation. It differs from and complements the other quality frameworks, rubrics, and scorecards in a handful of key ways:

• A courseware-specific resource, it addresses the unique product features and implementation considerations for courseware solutions of different forms, but doesn't endeavor to address all of educational technology.

• It establishes links between product capabilities and peer-reviewed research on learning outcomes, helping to build transparency into the learning science behind courseware product design.
• Its modular design allows users to access only the components that meet their needs. Over time, we envision subbing in comparable frameworks for components of the CWiC Framework, allowing users to work with their preferred frameworks in complementary ways.
• Freely and broadly disseminated, the CWiC Framework is offered under a CC-BY license, allowing users to share and modify the framework to meet their needs. Going forward, a committee of postsecondary leaders will drive the governance and maintenance of the CWiC Framework, releasing updates annually.

Downloadable and interactive versions of the CWiC Framework can be found online.

The CWiC Framework Explained

We designed the CWiC Framework to help stakeholders from across teaching and learning, research, and technology advance their courseware discovery, implementation, and development efforts. To serve these different audiences, the CWiC Framework includes four modular components bundled into three different instruments (figure 2).

The three instruments are:

• The CWiC Product Primer: This abbreviated tool helps users identify priority courseware capabilities during the product exploration and evaluation phase of selection. It helps diagnose your use case with one question and suggests a set of product features to look for in a courseware product based on that use case. This primer is ideal for faculty beginning to explore courseware solutions.
• The CWiC Designer: This resource is designed to support deeper understanding of a courseware product and the learning science principles that underpin its features. At its core, the CWiC Designer includes 45 "yes or no" questions about your courseware product features that support teaching and learning, and several product considerations for courseware procurement and delivery. It is ideal for instructional designers completing a more thorough review of a courseware product and may be useful for informing future product selection. It focuses only on product-related dynamics.
• The CWiC Framework: The complete framework consists of the 45 questions and tags to efficacy research included in the CWiC Designer, plus 20 questions on your courseware implementation. It is ideal for administrators completing course reviews and focuses on both product- and implementation-related dynamics.

Helping Courseware Achieve its Potential

Thanks in large part to the input of the development team's institutional advisory committee and dozens of courseware vendors, several important lessons shaped the CWiC Framework and have the potential to advance courseware use at individual institutions as well as in the field more broadly.

• Courseware quality can't be evaluated separately from the context for courseware use. Factors like the goals and environment for use of courseware are critical to selecting the right product for an individual course, department, or institution. Institutional leaders may look to institutions or courses similar to their own for recommendations on teaching and learning tools. It is also why we built the CWiC Framework to provide information on courseware quality based on context, rather than an objective quality score. Takeaway: To find a high-quality courseware product and improve the likelihood of a positive experience with courseware, it's important that you can answer this question: "What am I trying to achieve by using digital courseware?"
• Courseware is dynamic, with products and use cases evolving constantly. Products should be evaluated regularly for fit, and metrics for evaluation should be flexible or regularly updated to ensure relevancy to evolving products and environments. With regular reviews, users will better understand their courseware capabilities and confirm that it provides the functionality they value in a user-friendly way. For this reason the CWiC Framework will be updated annually to ensure relevancy to the latest products and use cases for digital courseware. Takeaway: If you're using the same technology RFP to guide your courseware selection, you're likely not asking some important questions that can affect your faculty and student experience using courseware.
• Significant gaps remain in the case for courseware's impact on learning outcomes, and filling those gaps is a challenge. Ongoing efforts from institutions, vendors, and research organizations have not yielded consistent data, leaving decision makers without a good rubric to measure courseware quality or effectiveness at their institution. In developing the CWiC Framework, we worked to identify existing research on learning outcomes and make it more accessible. Going forward, we hope that evaluations of courseware will be improved by using the detail included in the CWiC Framework to track and compare conditions in research studies. Takeaway: By using a common language for product and implementation inputs in courseware pilot studies and evaluations, we will be better positioned to understand what really moves the needle on student outcomes.

With broad adoption and use of the CWiC Framework, we hope to help bring down the language and knowledge barriers that complicate and impede courseware selection, implementation, and evaluation. Alongside the continued improvement of courseware products and ongoing efforts to establish an efficacy research base, this shift should change perspectives and practices around digital courseware use — creating conditions for courseware to achieve its full potential as a game-changer for students, faculty, and institutions.

Notes

1. Pubtrack is a subscription-based database. Tyton Partners pulled the same data for 2011 and 2014 to obtain the information referenced in this article.
2. Development of the Courseware in Context, or CWiC, Framework was funded by a grant from the Bill and Melinda Gates Foundation.

---

Gates Bryant is a partner with Tyton Partners.

Emily Lammers is a principal with Tyton Partners.