Competency-Based Education: Technology Challenges and Opportunities

Competency-based education (CBE) has elicited strong interest among educators and education stakeholders due to its potential to meet students where they are in their education journey and provide a more personalized path to completion. A typical CBE program has a curriculum structured to demonstrate learning in clearly articulated competencies, is often self-paced, is agnostic as to the source of learning while maintaining clear and transparent learning standards, and has an emphasis on authentic assessment, which evaluates what the learner knows and can do through real-life demonstrations and projects.

Unfortunately, the model's practical benefits are tempered by the significant technology challenges and barriers to CBE program adoption, roadblocks due to limits in (and among) higher education software products. As a community, the products used to manage our institutions and their teaching and learning processes are all deeply rooted in a credit hour–based, term-based, and course-based educational delivery model. CBE is not a new idea,¹ but its promise as a potential solution to critical issues in higher education (including affordability, completion, and transparency of learning outcomes) has stimulated renewed interest and significant growth, a trend forecast to continue.² But CBE growth is in fact stymied, along with its benefits to many of today's students, due a lack of software support.

In the early days of the CBE transition, suppliers (as a group) did not respond quickly and aggressively to the need for different features to manage CBE processes. This might now be changing, as this article explains. It is still too early to declare the problem solved, but there is clear evidence of positive action underway. Through the efforts of a partnership between the Competency-Based Education Network (CBEN) and the IMS Global Learning Consortium, the tide seems to be turning, and a roadmap for effective IT supports for CBE is taking form; this change is exemplified through the Technical Interoperability Pilot (TIP) project.

Background of TIP

The TIP project was founded by and for the C-BEN. The project's origin dates to the EDUCAUSE Annual Conference in October 2014 and a meeting between the C-BEN Business Process and Systems Committee (BPS) and a group of five product vendors to discuss the lack of systems support for CBE. (The BPS committee focuses on operationalizing CBE best practices in member institutions, at scale, and software systems are a critical component of that work.) At the time of the meeting, enthusiasm was continuing to build for CBE as a highly effective education model for many of today's college students, but systems (from learning management to customer relationship management, student information, and financial aid delivery systems) were not keeping pace with the growing demand. The C-BEN leaders at the meeting were senior administrators at CBE institutions and had personal experience trying to "fit" their CBE programs and students into systems and workflows designed for the credit-hour model. Even relatively simple tasks like enrollment and grade reporting seemed to require a major effort, and more complex work, like awarding financial aid, was nearly impossible without armies of staff to hand-manage workaround records and processes. Systems did not support CBE well, if at all, and to add insult to injury, the systems could not exchange basic data critical to managing CBE. Suffice it to say there was tension at the vendor meeting, which was called to ask, "What's going on? Why are you (and the market) not providing better products to support CBE"?

Listening patiently, one of the vendors said what in hindsight now seems obvious:

"Do you want to know the problem? There are no standards. No two schools manage their CBE programs the same, and there is no standard definition of a competency. Moving forward aggressively is too risky. Suppose we invest millions into one approach, and we get it wrong?"

In that moment, the C-BEN leaders and a group of CBE supporters present realized that action was needed to help break this standoff. The vendor perception that CBE programs had nearly infinite variation was unproductive — and wrong. While it is true there are differing CBE models and alternative program designs, these variations are not infinite. There are, in fact, a finite and limited set of program options relevant for vendor solutions. We could now see clearly that we had to document those options for the vendor community, to create a "CBE Profile" of the 35 institutions comprising C-BEN.

Conducting research and creating a document representing high-level CBE requirements would not be enough to stimulate action and jump-start the necessary momentum. CBE programs were up and running, and institutions needed help now. We needed vendor engagement on solutions, and the solutions needed to work together seamlessly to support the scale and operational integrity required to sustain accreditors' and regulators' scrutiny.

Interoperability Is Essential

Seamless interoperability is something that we have come to take for granted in the traditional credit-hour world. Today, thanks largely to the IMS Global Learning Consortium, concern over the transmission of data and content between platforms is low. Of course, this results from significant effort and time, spanning decades. In today's IT back office, interoperability problems are mostly the exception, not the rule. This unfortunately is not the case, however, at institutions implementing CBE. Basic information that needs to be exchanged for CBE is not represented in standards like LIS (Learning Information Services), Common Cartridge, and QTI (Question and Test Interoperability). To succeed in our efforts, we needed IMS Global to be engaged with TIP at the outset, and they have been a committed and critical partner from day one.

Therefore, the TIP project would have two halves — a research half and a "solutions" half. The research half would uncover common and critical CBE technology issues, and the solutions half would engage vendors to develop interoperable solutions. The solutions would not necessarily be production-level software, since it is wildly unrealistic to believe we could hijack the product roadmap of a software company, especially for a voluntary and short-term project. But the prototype solutions would address real problems and begin to develop critical momentum. What incentive would a vendor have to volunteer valuable product development resources to develop software, even a prototype? We hoped the answer was market information. The C-BEN schools are important as individual institutions, but 35 institutions represent a wide swath of a growing market. The institutional, program, and operational data that TIP assembled, quantifying the problems and the contours of the CBE programs, could offer valuable insight to a product company looking to anticipate and address an emerging need.

The TIP project launched officially in January 2015. The project's mission statement: Speed the delivery to market of quality CBE-supporting software, serving as a market accelerator for interoperable solutions supporting CBE program delivery. TIP will conclude as a C-BEN project in December 2015, but the collaborative efforts ignited by TIP will continue and grow beyond the project's end date.

TIP Research

To understand the common technical challenges faced by member institutions, and therefore to guide the selection of pilot solutions for the project, we created a research survey with two sections: Systems in Use, and Technical Issues. In the first section, respondents answered questions regarding the software products and versions they used to perform common enterprise functions. The TIP Reference Architecture (figure 1) was provided to guide the responses and promote comparability between institutions. Thanks in great measure to the staff at Public Agenda who administered the survey and to the C-BEN schools themselves, we had a survey response rate of 72 percent. 

System Satisfaction for CBE

In addition to simply inventorying the systems (vendor products) used, the survey also captured satisfaction levels on a five-point scale. The satisfaction question asked only about CBE programs (not credit-hour programs). Finally, the survey queried the degree of customization for each application category on a scale ranging from little or no customization (level 1) to highly customized (level 5). Completely custom-developed applications were designated as such. See table 1.

Table 1. Satisfaction levels by category

System Category	Satisfaction
Student Relationship Management	3.33
Student Information System	2.87
Curriculum Management	3.05
Financial Aid	2.92
Student Success	3.17
Assessment Administration	3.78
Learning Management System 1	3.09
Assessment Tools	3.44
Teaching and Learning Tools	3.50
Instructional Materials	3.45
Eportfolio	2.94
Data Repository	3.32

 

We recognize the limitations of this survey in terms of its relatively small sample, sampling method, and other biases. However, for TIP purposes this data helped guide the selection of pilot solutions.

Technical Issues

The Technical Issues section of the survey consisted of 27 questions designed to be answered yes or no. This highly guided and prescriptive method has clear limitations, but it also allowed us to learn about the common technical issues quickly. The subsequent interview process allowed for a more nuanced understanding of those issues and therefore a more informed analysis of which of the big-ticket problems we should address.

At the headline level, the survey clearly showed that the concerns and experiences of the C-BEN leadership at the vendor meeting were not isolated but broadly representative of the institutions they represented. Nearly 80 percent of respondents said there were significant technical issues or inefficiencies managing their CBE process or information with their existing systems (figure 2). Now, what were those problems, and which ones could we begin to resolve with TIP?

Data Management Deficiencies

Imagine trying to manage the course catalog (internally and on the web), enrollments, grading, and transcript processes without a course number or other identifier to uniquely track and manage a course. Next, see what happens when the course description changes to a new version, again with no unique code or identifier to keep a change history. It would result in havoc, of course, but that's the unfortunate situation for fully half of the CBE program administrators and their colleagues using software products that do not manage competencies well. At this stage, in many systems, competencies are not managed in the same way as courses, with a unique competency code and related competency statement that can be added or changed and removed as needed (figure 3). How will we perform analysis on the effectiveness of a CBE program if what we want to measure is not managed in an integrated database? Managing competencies is therefore an important issue in search of a solution.

Course-Competency Relationships

Courses are the educational construct that we know, a "container" for offering and assessing knowledge. Competencies are discrete demonstrations of learning. In the survey we asked three questions to understand a critical data management subject — the relationship between courses and competencies (figure 4). With the exception of a few "direct assessment" schools that are experimenting with programs without a course construct, we will have courses to manage for a long, long time. So understanding how competencies relate to institutional and program objectives and to courses and course modules is critical to managing competencies and exchanging data between systems. Therefore it is also critical to vendors developing software.

From this and other question responses it is clear that while competencies are related to courses, competencies are not courses. Competencies can be a component of a course, and competencies can span courses, as we found in 50 percent of the cases. What this means for data analysts and designers is that courses and competencies have a many-to-many relationship. This is one of the essential lessons for vendors — to design their databases to address the myriad of CBE programs available today.

Competency Scores Held Hostage

One of the more interesting results from the survey is that LMSs as a category are judged "moderately satisfactory." But a deeper look into the data tells a useful story about the value of interoperability. Getting to this conclusion requires stacking two survey questions next to each other, and Dr. Pareto himself would be proud of this near 80/20 relationship (figure 5).

Nearly 75 percent of responding institutions capture CBE-based assessment results in their LMS gradebook. But there the score stays, for all but a handful of schools that probably have built a customized solution. LMS to SIS grade reporting has been a common feature available in a standard way through IMS's LIS, but LIS does not support competencies and course-competency relationships. This seemed like a problem that could be solved with a prototype solution.

"An Issue at Every Level"

This quote came from a respondent asked about the self-paced nature of CBE — easily the most structurally significant conflict with the traditional academic calendar and processes faced by CBE adopters (figure 6). Not all self-described CBE programs in operation today are self-paced, perhaps precisely for that reason. Federal Student Aid regulations assume credit hours within the traditional semester-based calendar, and multiple regulatory conditions are tied to those assumptions.

Thankfully, the Department of Education offers a relief valve called the "non-term" calendar. Effectively, the student's enrollment starts the schedule, and their planned project completion date defines the end point of the aid period. Although not perfect, it's workable — but challenging to implement. The two largest SIS providers (based on C-BEN respondent usage only) do not support non-term financial aid calendars, yet 64 percent of respondents use their SIS as their financial aid system. Schools need a choice of financial aid providers, whether a third-party service agency or a software product designed for non-term financial aid. Either way, the service or system will need SIS data to award financial aid correctly. A standardized data exchange for non-term financial aid will help speed CBE adoption and solve this important and common issue.

Measuring Faculty-Student Interaction

Another issue related to financial aid but also of significant value more broadly is the subject of "regular and substantive interaction." Everyone wants to ensure that federal student aid dollars go to students actively engaged in learning, and an overwhelming number of responding schools track this engagement measure, but those records are mostly manual and often spread over numerous source files, systems, and journals. Thanks to a new standard from IMS Global called Caliper, there may be a way to capture those faculty-learner touchpoints in a consistent format ripe for analytics — regardless of where the interaction occurred (figure 7).

Transcripts

As institutions begin to add CBE programs to their offerings, they will quickly confront the challenge of reporting academic performance through the traditional transcript. Competency programs can provide a rich and descriptive understanding of what the learner knows and has demonstrated — capabilities of keen interest to students and prospective employers. The traditional systems of record (the SIS) are mostly incapable of producing such reports, and schools are resorting to spreadsheets, word processors, and other sources like their LMS to produce a CBE transcript (figure 8). These workarounds will have dubious quality assurance mechanisms and will complicate data and records administration. A single integrated student record has contributed to administrative quality assurance for decades in higher education systems; significant forces now challenge the primacy of the SIS as the system of record.

Solutions Phase

One purpose of the TIP project was to identify common issues that meet the "80/20" rule, in that they are relatively small and doable, and would have an outsized benefit if resolved. Based on the research findings, the TIP team identified five use cases that met this requirement:

1. Managing competencies using a unique key, in an integrated database, including the course-competency relationship(s)
2. Reporting assessment evaluation results between an LMS or assessment platform and the SIS or system of record
3. Extracting CBE program information for non-term–based financial aid
4. Measuring components of regular and substantive interaction
5. Producing an extended CBE e-transcript

Partnership with IMS Global Learning Consortium

Armed with this short list of solutions to model and prototype, the TIP team set out to match the issues reported by the C-BEN survey respondents with vendors serving those specific schools — in search of volunteer vendors willing to work in technical solution teams facilitated by IMS Global. Twelve vendors stepped forward to participate, and this group has been designated TIP Participating Vendors for their significant efforts to improve support for interoperable CBE for their customers and their customers' students.

Of the five use cases selected, one case involved unexplored territory for the incumbent C-BEN vendors — using technology that was new to them and therefore requiring a greater resource commitment. As a result it was necessary to supplement the original list. Fortunately, IMS Global contributing members stepped forward to fill the specialized needs of the digital credentials project (use case 5) and the use case to manage competencies (use case 1).

C-BEN vendors/TIP participants (alphabetically):

• Blackboard
• D2L
• Ellucian
• eLumen
• Flat World
• Oracle
• Pearson
• Regent Education
• Vital Source

Supplemental TIP participants (alphabetically):

• Accreditrust
• Learning Objects
• Workday

In the spring, work began with the vendors working in three teams related by their assigned use case(s). The teams were led by Joellen Shendy of the University of Maryland University College and Jeff Grann of Capella University (both schools are IMS contributing member institutions), with the architectural guidance of John Tibbetts, IMS chief product officer. The teams met in small groups and worked through the conceptual models and diverse implementations of competency programs to tease out the commonality and ground truth of competency-course and other data relationships, in order to build a prototype data model and associated natural data "rules." From those efforts a design of a prototype digital credential (e-transcript) was created, and a prototype reference data model was produced. Participating vendors then used the reference model to evaluate their own proprietary CBE implementations to ensure that they can exchange data with other systems in a way that is consistent and interoperable, and to develop prototype data exchange schemas for use cases two through five, which involve data hand-offs between systems (figure 9).

Progress Report

The purpose of the TIP project was to speed the availability of interoperable CBE solutions for C-BEN schools and the market in general. Through the facilitation of IMS Global and its members, institutions and product suppliers are working together on a common vision for CBE interoperability, based on a developing set of common concepts and understanding.

Progress has been encouraging if uneven, with each of the 12 participating vendors starting at different places, having their own visions and roadmap priorities, schedules, and challenges to contend with — but progress there has been. Critical mass has been reached, and that all-important momentum has been built and is sustainable through IMS Global's working group protocols. The all-volunteer TIP-CBE team has:

• Researched numerous published work products and incorporated many of their ideas into the standards data model
• Formed a task force of registrars following  AACRAO guidelines to explore the strategically important intersection of academic records and digital credentials and develop a prototype for a secure, extended transcript for CBE with components freely available in open source
• Developed an initial data model to support five use cases to manage competencies and provide interoperability between SIS, LMS, Curriculum and Assessment Management, and non-term financial aid products or services
• Developed prototype integrations between volunteer LMS and SIS suppliers to report competency evaluation results to a system of record
• Informed the community of TIP research findings and the technical collaboration efforts through presentations and published works to continue to build momentum for CBE technology support

This is very good, even remarkable, progress considering the scope of the effort and its volunteer staffing nature, but it's not enough. There is more to do. Hopefully, this article becomes one of many progress reports shared on the road to complete implementation of competency-based education as a fully supported model, perhaps the model of education for the 21st century.

Acknowledgments

Recognition and thanks for their contributions to this project go to: The member Institutions of the Competency-Based Education Network and its leadership; the team at the IMS Global Learning Consortium led by Rob Abel; the team at Public Agenda led by Alison Kadlec, and especially Erin Knepler for her survey support; to Stacey Clawson of the Bill and Melinda Gates Foundation for sponsorship of TIP, and to Kevin Corcoran of the Lumina Foundation for his vision and support for the C-BEN community of practice and related efforts. Appreciation for review, editing, and guidance in this and other TIP work goes to Deb Bushway, on assignment as senior policy advisor to the undersecretary, US Department of Education, and to Charla Long, former founding dean of Lipscomb University's CBE program and a consultant to Public Agenda. Thanks to Joellen Shendy, UMUC registrar, C-BEN and IMS Global team leader, and Jeff Grann, academic director of Assessment and Learning Analytics at Capella University, and to John Tibbetts, IMS Global’s chief product officer for their herculean efforts. Finally, a special thanks to Mike Offerman of Offerman Consulting for his wisdom, guidance and support throughout.

Notes

1. Kate Ford, "Competency-Based Education: Its History, Opportunities, and Challenges," UMUC Center for Innovation in Learning and Student Success," October 24, 2014.
2. Brian Fleming, "Mapping the Competency-Based Education Universe," Wake-Up Call, Eduventures, February 17, 2015.

---

Mark Leuba is managing director at Pathway Technology Partners, TIP Project Leader, and IMS Global CBE facilitator. Formerly CIO for American Public University System, he led the technology team through many challenging growth milestones such as regional accreditation, Title IV access, and its initial public offering. At Pathway Technology Partners, he recently contributed to projects for the Bill and Melinda Gates Foundation. Mr. Leuba holds a BS degree in Business Administration/Finance from the University of Baltimore and an AA in Information Technology from the Community College of Baltimore County.