Despite a late start, established and start-up competency-based education vendors have begun to provide technical tools to support CBE. This update on the Technical Interoperability Pilot project reports on progress developing interoperable solutions and offers cautionary advice to the CBE institutional and vendor community: avoid complacency and press on, quickly.
This article recaps the latest and final planned results of the Technical Interoperability Pilot (TIP) project, a research and development initiative funded by the Bill and Melinda Gates Foundation and originally introduced in fall 2015 in this publication. The TIP project, a collaboration of the Competency-Based Education Network (C-BEN) and IMS Global Learning Consortium, has the goal of speeding interoperable CBE technology support to the higher education market.
Interest in Competency-Based Education
Interest and enthusiasm for CBE as an important educational model continues unabated, with an estimated 586 institutions currently operating or planning CBE programs according to research from the CBE Landscape Project's CBExchange.1 However, technology application support for these programs lags far behind the increasing demand. Inadequate technology for CBE presents significant barriers to operation among 79 percent of C-BEN institutions.2 While some better-funded schools invest in custom solutions to address the gaps, this is not an affordable or advisable path for most institutions. More and more products have CBE-friendly components but lack fully integrated solutions.
Importantly, in the past year numerous vendors large and small made a full-scale commitment to CBE interoperability in their products through collaboration with IMS to define interoperability specifications. This article summarizes the TIP pilot program and improvements in CBE technology.
Forecast for Change
TIP has partnered with dozens of institutions in this work, and the most common question is, "How long before we have the systems we need to support CBE?" The signs point to a 12–18 month window for many products, with vendors announcing road maps such as "summer 2016" for important releases that will include key features for CBE — but upgrade cycles take time to plan, and target slippage occurs. Other vendors have specific CBE-supporting features "currently available." Let the buyer beware, however: understanding the process lifecycle is critical to appreciating and planning for all of the discrete components that must be in place and aligned to fully support a capability that might exist in one vendor's product or module. Since few processes stand in isolation, vendors must continue to engage around IMS's common specifications for CBE data interoperability in order for their systems to exchange data productively.
Unfortunately, not all vendors plan to update their existing products for CBE. Some have chosen to wait for their next-generation products, leaving existing customers to either wait or search for alternative niche solutions that will integrate with their existing platform. Since new platform migrations can be complex affairs, the timeline for those lagging products will likely be 18 months plus. Institutions should press their vendors to provide demonstrations of their CBE capabilities in real-world scenarios. They should not rely on generalizations or broad assurances about CBE being supportable by features not designed to support competencies as full-fledged entities in the product's integrated database. Vendors frequently respond to market signals, and schools can continue to work together in technology forums like IMS Global and EDUCAUSE to speak with a loud and coordinated voice. This is one way to pressure software vendors to deliver on interoperable, tech-enabled CBE.
Technical Interoperability Pilot Project
The TIP research, described in detail in the introductory report, called out urgent issues involving CBE support, with 79 percent of research participants reporting significant technical issues or inefficiencies managing CBE processes with their internal systems. Many of these issues arise from deficiencies in managing the competency data within and between systems. In fact, most application databases lack basic elements of data management for CBE such as unique record identifiers for competency statements.
Data Management Deficiencies
At this stage of CBE development and adoption, in many institutional systems competencies lack a "competency code." As a result, systems cannot manage competencies in the same standardized way as the do courses, each of which has a unique identifying code and related description that can be changed or removed as needed. Addressing this gap in curriculum and student databases is job one in establishing a foundation for CBE data management and analytics. To prototype an interoperable solution for this issue, the TIP project and IMS Global teamed up with Ellucian and Workday in Use Case 1 — Managing Competencies.
A note about the solution prototypes: Each of the prototypes was of limited duration — 120 days, which is a short period in software design and development cycles. Therefore the expectations for the prototype emphasized the vendor's understanding of the issue and their solution design approach, not actual production code. Another evaluation criterion was the vendor's intention to implement the solution in a future release. A survey asked the institutions to rank the prototypes (see the sidebar).
Institutional Survey
The brief survey form asked the institutional evaluators to rank the following six subjects on a 1–5 scale (1 = poor, 5 = excellent):
Understanding: Did the vendor understand the problem or issue to be solved? Was the vendor working on the right problem as described in the use case document?
Approach: Does the vendor's approach address the issue in the use case document? (Answer whether or not the prototype is "complete." Vendor's progress will be at different stages, and few if any will be "in production.")
Usefulness: If the vendor were to implement their solution in production, will it be useful? Will the solution help solve a real problem?
Gaps in Approach: Are there obvious needs the vendor's vision does not address relative to the use case as described? Understanding that a prototype is intended to be a rough start, are there still obvious areas of concern that have not been considered?
Direction: In general, is the vendor headed in the right direction with CBE? Considering that the prototype may be a first step, do you believe the vendor(s) are headed in the right direction to address the broad issue in the use case?
Product Plans: Is the vendor planning to implement the use case in their product? Vendor product cycles are sometimes planned years in advance. Will this solution (or one like it) be in their product roadmap?
For each category in the survey the evaluators were encouraged to provide explanatory comments and suggestions for improvement that would be shared with the vendors.
Evaluators' rating of the prototype for Use Case 1 — Managing Competencies
|
Understanding |
Approach |
Usefulness |
Gaps |
Direction |
Product Plans |
|---|---|---|---|---|---|
|
5.0 |
4.5 |
4.5 |
4.0 |
4.5 |
4.5 |
Scale of 1–5, with 5 = excellent
Exchanging Competency Evaluation Scores
In the TIP research, the evaluators judged learning management systems as "moderately satisfactory" in their support for CBE. But digging into the data identifies a barrier to CBE scalability. The following research questions tell the story: Institutions are using their LMS's to capture competency evaluation scores, but the scores are not automatically exchanged with their student information systems and transcripts — they sit in the LMS gradebooks.
- Do you capture competency evaluation scores in a gradebook or equivalent module in an LMS? Yes = 71 percent
- Are competency scores automatically transferred from the system of initial capture to the student information system? Yes = 17 percent
Lacking a scalable way to collect competency evaluation results, the scale and quality of CBE programs will continue to suffer from the manual effort needed to enter and re-enter scores. Although IMS Global's interoperabilty standard Learning Information Services solves this problem for credit hour programs, LIS requires updating for CBE. The prototype Use Case 2 — Evaluation Results would help identify the key requirements for updating the next generation of the LIS standard.
C-BEN institutions use Blackboard and D2L Brightspace as their number 1 and 2 LMS providers, respectively, and those two suppliers collaborated through IMS Global in the development of the Use Case 2 prototype data exchange.
Evaluators' rating of the prototype for Use Case 2 — Evaluation Results
|
Understanding |
Approach |
Usefulness |
Gaps |
Direction |
Product Plans |
|---|---|---|---|---|---|
|
4.7 |
4.0 |
4.7 |
3.7 |
4.7 |
5.0 |
Scale of 1–5, with 5 = excellent
Issues with Self-Paced Programs and Financial Aid
"An issue at every level" is a how a TIP research respondent described the self-paced CBE programs using tools designed for fixed term academic calendars. The two largest student information system providers (based on C-BEN TIP respondents' usage only) do not support self-paced (called "non-term") financial aid calendars, yet 64 percent of respondents use their SIS as their financial aid system. As one respondent noted, "The workaround is massively labor intensive." Schools need a choice of financial aid providers, and the external service or system will need SIS data to award financial aid correctly.
To prototype a solution to this issue, two C-BEN suppliers collaborated with IMS Global: Oracle with their PeopleSoft Campus Solutions SIS and Regent Education, a provider of financial aid software. They teamed up to prototype Use Case 3 — Program Information for Non-Term Financial Aid. Oracle served as the host for program and student information, exported in a standardized form to Regent for calculations and processing.
Evaluators' rating for the prototype Use Case 3 — Program Information for Non-Term Financial Aid
|
Understanding |
Approach |
Usefulness |
Gaps |
Direction |
Product Plans |
|---|---|---|---|---|---|
|
5.0 |
4.3 |
4.3 |
4.0 |
4.3 |
5.0 |
Scale of 1–5, with 5 = excellent
Measuring Faculty-Student Interaction
Another issue relevant to program quality overall and financial aid decisions involves measures of regular and substantive interaction between students and faculty, as well as students and other staff. The majority of C-BEN responding schools track these contacts. Their efforts are mostly manual, however, and often involve numerous source files, systems, and journals. A new standard from IMS Global called Caliper offers a way to capture those faculty-learner interactions in a consistent form ideal for analytics.
Working together, IMS members eLumen and Flat World interviewed C-BEN institutions to learn more about the evolving regulatory regime and how institutions track and report these measures. The resulting prototype design for a new Caliper-based "Engagement Metric Profile" is the result of Use Case 4 — Measures of Faculty-Student interaction.
Evaluators' rating for the prototype Use Case 4 — Measures of Faculty-Student Interaction
|
Understanding |
Approach |
Usefulness |
Gaps |
Direction |
Product Plans |
|---|---|---|---|---|---|
|
4.5 |
4.0 |
4.0 |
4.0 |
4.5 |
3.5 |
Scale of 1–5, with 5 = excellent
Providing Extended Transcripts
Another use case emerging from the TIP research relates to digital credentials for CBE: Use Case 5 — An Extended Transcript for CBE. A survey indicated that 42 percent of C-BEN institutions are issuing "dual transcripts" — one or two documents that include traditional credit hour grade measures and the new CBE mastery-based achievement designations. Some institutions (30 percent) include defined levels of performance beyond "mastered." As of yet, the information required to produce this transcript is most often not housed in the traditional system of record, the SIS, but in the LMS. For expediency's sake, institutions have compromised the rule that the SIS is the system of record for most, if not all, student academic records because of the time and costs required to customize the SIS to store the required competency information. This problem underlies many of the institutional issues affecting CBE programs and planners. In response, the market has moved to provide enterprise-scale software as an alternative to the SIS for student records.
The need to develop an alternative transcript for CBE presents both problems and opportunities. In the connected age, where virtually all aspects of life are moving to the web, the transcript is assumed to be digital, an e-transcript. If we begin with the assumption of an e-transcript, the other forms and uses we historically have employed can readily be accommodated (e.g. printed copies or PDF, or XML data exchanges); at the same time, new and better uses can now be supported. With a standards-based e-transcript, now-routine web practices can provide security and trust verification and link to a web of data about the institution, the program, evidence of learning, and more. Work to foster standards-based digital credentials may enable a future where the learner's secure digital record can become a highly valued personal asset.
In response to the need for an extended transcript for CBE as documented in the C-BEN research, IMS Global has sponsored a prototype project to create a standardized e-transcript file format and reference e-transcript generator (available as open source). See figure 1. The eT project adopts the JSON-LD W3C standard, which lays the groundwork for many future web-based features and benefits, including compatibility with the Open Badge standard and the envisioned Credentials Transparency Initiative.
Figure 1. Extended transcript prototype example (IMS Global)
Acknowledgments
Without the support and participation of C-BEN institutions and their volunteers, the TIP research and development project would not have been possible. Special recognition is due to the institutional reviewers for their help providing feedback and guidance to the vendors on their prototype work: Annie Myers, Broward University: Laura Kite, University of Wisconsin Extension; Bridget Gaer, Capella University; Jon Sizemore, University System of Georgia; Brian Miller, Davenport University; Brendan Farley, Walden University; Mary Hendrix, Texas A&M University; and Julie Uranis, Western Kentucky University.
Notes
- Public Agenda, "A Research Brief on the Survey of the Shared Design Elements & Emerging Practices of Competency-Based Education Programs," December 2015.
- Mark Leuba, "Competency-Based Education: Technology Challenges and Opportunities," EDUCAUSE Review, October 12, 2015.
Mark Leuba is vice president, Product Management, for IMS Global, a leader in education technology interoperability, where he guides the development and execution of a cross-functional, integrated product plan anchored by breakthrough standards such as LTI and Caliper. Prior to his work with IMS, Leuba held positions of executive technology leadership in higher education, commercial ed-tech platform companies, and private consulting practice to leading foundations and institutions.
© 2016 Mark Leuba. This EDUCAUSE Review article is licensed under the Creative Commons BY 4.0 license.