Digital Badging in the MOOC Space

Early adopter universities have a number of strategic aims for massive open online courses (MOOCs), including advancing education research, improving access to education, and demonstrating their merit.¹ One area of education research of particular interest to the University of Notre Dame is that of digital badges, which are "visual symbols of credentials...[that] make specific claims about learning and...are designed to be shared easily."² These representations of achievements or skills are more granular than a formal degree and help make incremental learning more visible.³ MOOCs have a notoriously low average completion rate (15 percent),⁴ which suggests that MOOC participants lack the incentives needed to complete the courses and master the material. A number of organizations, including Notre Dame, are exploring the various uses of digital badges and how digital badges can improve learning online and on campus, including in MOOCs. In addition, the latest evolution of MOOC credentials is moving toward (edX) micro-masters and (Coursera) specializations.⁵ These series of MOOCs concentrate on a particular topic and provide an alternative to a traditional degree. MOOC micro-masters and specializations are usually free, with just-in-time learning at a fraction of the cost and time of other programs. To date, few MOOCs have issued digital badges. Two early projects include, Dartmouth's Introduction to Italian Opera and George Washington University's Practical Numerical Methods with Python.

The goal for the Notre Dame pilot was to iteratively study the design and implementation of digital badges in the global MOOC environment using three edX courses. In the first MOOC, I "Heart" Stats: Learning to Love Statistics, the research team explored the impact of badges on course completion. In the second MOOC, Math in Sports, the researchers evolved their exploration to examine the profiles and perceptions of badge earners. In the third MOOC, The Meaning of Rome: The Renaissance and Baroque City, the research team wanted to dig deeper and uncover the story — and evidence of value — behind the badge.

Case 1: What Impact Does the Badge Have?

In early 2015 Notre Dame launched its first MOOC, I Heart Stats. The course aimed to help students learn how to select appropriate statistical tests for data, perform basic calculations to determine statistical significance, use standard methods of representation to summarize data, and interpret and assess the credibility of basic statistics. I Heart Stats participants also had the option to earn the standard honor code certificate or verified certificate through edX. To earn the honor code certificate, the course participant had to complete all of the course modules and earn at least a 70 percent on all course assessments. (Note: edX no longer offers this free honor code certificate.) The verified certificate had the same criteria but also required the participant to pay $50 to have his or her identity verified. Notre Dame designed, developed, and implemented a free digital badge via Credly (see figure 1) that had the same criteria as the honor code and verified certificates, was attached to the badge earner's identity, and was also connected to and verified by Notre Dame. Of the enrolled students, 151 learners claimed the Notre Dame course completion digital badge.

To measure effects of the digital badge's availability on course completion, we compared pass rates across two groups:

• A = students chosen by random sampling to be offered the digital badge (N = 3,651)
• B = students randomly selected not to be offered the digital badge (N = 3,651)

A total of 868 participants passed the course, with 438 (or 50.04 percent) in the A group and 430 (49.96 percent) in the B group. The A group did not have a substantially higher pass rate than the B group. This further demonstrates the fallacy that a badge offer alone would encourage course completion. Table 1 shows the differences among the A learner groups who were offered a badge and those who expressed interest in receiving the offered badge. The subset of A group students interested in the digital badge had a 30.8 percent pass rate compared to the overall 12.1 percent pass rate of the entire A group. Furthermore, 329 of the 438 (75.1 percent) course passers in the A group who were eligible for earning the badge expressed interest in a week 2 email survey. Though the badge did not affect the total number of course passers in the A group, these observations suggest that interest in earning a badge could be an early indicator of commitment to and achievement in a course.

Observation: Although availability of the digital badge did not significantly influence overall MOOC course pass rates using the A/B test, a substantial increase in pass rate occurred in the A group learners interested in badges (30.8 percent) compared to the overall A group. This information could help MOOC faculty identify serious learners on a successful pathway earlier than previously possible. By figuring out who the early engagers are (those interested and motivated to earn a badge and complete the course), MOOC providers could give priority to the committed students and perhaps allocate an extra chunk of limited resources to provide them additional discussion feedback.

Table 1. Comparing pass rate among learning groups

Group	No. of Passers/N	Pass Rate
All MOOC participants (total enrollment)	868/20,206	4.3%
No badge offer (B group)	430/3,651	11.8%
Badge offer (A group)	438/3,651	12.1%
Interested in badge (in A group)	329/1,067	30.8%

Table 2 shows the credentials that MOOC passers (from the A group) were interested in. These data reveal that 278 of 384 (72 percent) total honor code certificate earners and 45 of 54 (83 percent) total verified certificate earners wanted to duplicate their credential as a digital badge. This suggests that something was missing from the existing edX MOOC free honor code or paid verified certificate credential; otherwise why would students want to replicate their certification in the form of a digital badge? Of course, that badge was verified, branded, and connected to Notre Dame, which might have appealed to students separately from MOOC certification.

Table 2. Credential interest by course passers in the A group

Earned	N
Only honor code	106
Only verified certificate	9
Honor code and digital badge	278
Verified certificate and digital badge	45

Case 2: Who Earned the Badges?

During the next MOOC, Math in Sports, the design team wanted to expand the previous "completion" digital badge to include peer-situated "achievement" badges that would have more social currency. In addition to the standard pass badge (figure 2), the team designed, developed, and offered two new peer-ranked badges to all MOOC participants. The "Coach" badge (figure 3) sought to recognize the peer leaders within the community of learners, specifically in the discussion boards. The "All-Star" badge (figure 4) aimed to identify those who had exceeded the minimum 70 percent passing score and passed with high honors, scoring 90 percent or above. Of those who claimed badges, 21 claimed the Math in Sports badge, 44 claimed the All Star badge, and 76 claimed the Coach badge.

Demographic information collected from course participants and a post-course survey allowed the researchers to determine that the majority of badge earners in this course consisted of formally educated males from the United States with little MOOC experience and a weak affiliation with Notre Dame. More specifically, a typical badge earner lived in the United States (40.5 percent) or Europe (30.85 percent; see figure 5), was male (84 percent), did not fit the age range of a "traditional" student (figure 6), and had at least a bachelor's degree (83 percent; see figure 7). These characteristics largely fit the profile of a typical MOOC participant.

The more interesting findings regarding the profile of badge earners actually come from the exceptions to this general profile. For example, only 26 percent of badge earners were female, yet females earned 46.5 percent of the Coach badges (figure 8). Likewise, a disproportionate number of All-Star badge earners live in Asian countries (see figure 5). Both exceptions suggest that badges can be designed to engage specific populations of learners, which could be particularly important for universities that hope to expand their global outreach by offering MOOCs.

The research and design team also wanted to determine participants' perceptions of the digital badges. To examine this issue, they asked direct questions on the post-course survey and made inferences based on participants' earn-vs.-claim badge behavior. Perhaps most importantly, the team concluded that digital badges should serve a role distinct from the verified certificate for course completers.

Observation: The claim rates for the All-Star and Coach badges — which reward specific, outstanding course behaviors — were more than double the claim rate of the Math badge, which rewarded completion.

Figure 9 shows the claim rates for the three badges offered. Participants also shared exceptionally positive comments on the post-course survey regarding the Coach badge:

• "The Math in Sports Coach Badge indicated my level of involvement (community orientation) in the course."
• "I was pleased to earn the Coach badge. I wanted to become more involved in forums this time as I had not participated quite as much in the other 4 MOOCs I took and felt I missed something by not doing so."
• "Nice to see a credential which validates the fundamental/organic effort of class participation that is the root of online academic efforts and their comparative value over time…"

Despite these positive findings, many of the responses on the post-course survey indicated that digital badges still generally meant little to course completers. Multiple survey respondents wrote that the digital badge(s) "meant nothing to [them]," and the substantial majority of those who did not pursue or claim a digital badge indicated that they did not need the badge or did not understand its value.

Observation: To increase the impact of digital badges, it is critical to improve communication of the benefits of badges for those who earned them.

Finally, of those who did claim the digital badges they earned, 40 percent indicated they plan to display their badge on a social networking site. Analytics from the badge-issuing site (Credly) indicate that the overwhelming majority (88 percent) of these social media displays take place on LinkedIn, compared to 11 percent on Facebook and one percent on Twitter. This suggests marketing future digital badge offerings for LinkedIn posting and technically integrating them with LinkedIn.

Case 3: Where Is the Story Behind the Badge?

The final design case took place in the edX MOOC The Meaning of Rome: The Renaissance and Baroque City. Beyond the typical paid verified (pass) certificate, this course offered the "Tempietto" badge (figure 10), an optional performance/project-based capstone challenge that gave students who passed the course with a 70 percent or higher the opportunity to build an e-portfolio demonstrating architectural concepts and skills from the course. In essence, this opportunity let students showcase their learning in a public, dynamic way — far more than the typical "watch videos and take a multiple choice quiz" for a MOOC certificate. The campus e-portfolio group worked with engineers from Academic Technologies and Digication (the campus vendor) to build a first-in-kind seamless integration into the edX platform, giving all MOOC learners an e-portfolio account connected to their edX profile. The MOOC learners then submitted their e-portfolios for review by the faculty and teaching assistants through Digication's back-end Assessment Management System. The evaluators used a rubric to evaluate the quality of the MOOC learners' e-portfolios. Badge earners were encouraged to make their evidence and badge public on their e-portfolio.

To answer this case's design question, the research and design team dove deep into the story of one badge earner. Alaa is an architecture student at Damascus University in Syria. She was one of thousands who enrolled in the course, one of a few hundred who completed and passed the course, one of around a dozen who took on the extended e-portfolio capstone badge challenge, and one of only six who emerged as a Tempietto badge earner with a passing rubric score.

One of the most obvious benefits of issuing badges to Credly profiles is the humanizing of MOOC students by allowing profile photos. Clicking deeper provides another level of significance: If you click on Alaa's profile in the list of people who earned the Tempietto badge, you will see the badge with its "baked in" metadata. Figure 11 displays the description, criteria, issue date, recipients, and most importantly (and usually under-used) the evidence field.

Again, because the design team integrated the e-portfolio into this MOOC, Alaa's evidence link takes you to her public e-portfolio. Figure 12 shows some screenshots of her pictures, drawings, videos, and reflections that show the story behind her course badge and evidence of her course performance.

While her country is being torn apart by civil war, Alaa states that she wants "to bring attention" to the ancient city where she lives, Damascus. The e-portfolio and the open badge empowered this young Syrian woman to use Notre Dame's free MOOC to demonstrate her interest in and competency with architectural analysis — using her home city as her canvas.

Observation: Open evidence of learning with an open badge means to be transparent, to give public visibility, to bring attention to the student and his or her skills through creation of an open e-portfolio.

At the center of open evidence of learning lie digital portfolios and badges. If paired right,they can unlock the power of evidence behind the open badge and optimize a student's ability to collect an available body of projects and the process to make and prove a competency claim. Of course there are multiple ways Alla could have gained these same knowledge and skills from the MOOC but… if Notre Dame didn't offer this free MOOC would Alaa have had this opportunity? Could we, or others, have known or seen Alaa, her work, her story, her learning, her architectural analysis of her ancient city without the e-portfolio? Here are links to a few other badger earner's public portfolios to see how they applied the course to St. Patrick's Cathedral (NYC), Dublin (Ireland), Chicago, and Salt Lake City (UT).

MOOC Badge Design Takeaways

Digital badging — still considered an emerging technology — will take more time to be adopted and valued at large scale. They may provide stepping stones to an unbundled, stackable, and more micro alternative credential with better assessments based on evidence of a skill. In these early years, digital badges face the danger of inflation if the main criteria for earning one is a "warm body." Peer-situated achievement and evidence-based performance can raise badge standards and increase their perceived value. Since 2013 we have observed MOOC and on-campus badging at Notre Dame. Simply counting the number of badges issued won't suffice. Instead, we wanted to know whether earners valued the badges enough to claim and display them. A learner in the MOOC needs to enroll, become aware and interested in badge opportunities, complete and pass the course, and then earn, claim, and display the badge.

Using the MOOC as a large-scale global sandbox yielded a few methods and metrics for designing and evaluating the value and adoption of digital badges:

• Examining the badge's impact beyond completion
• Understanding the importance of the badge user profile
• Measuring badge user attitudes and motivations
• Looking closely at the level of badge user engagement (earn –> claim –> display)
• Examining the evidence and qualitative story beneath the badge
• Defining the types and purposes of MOOC badges: completion, achievement, and performance

Figure 13 shows the evolution of Notre Dame's MOOC badges from completion (pass) to achievement (peer-ranked) to performance (evidence-based).

Over the three iterative design cases of developing and evaluating five different digital badges granted to 233 global learners, we learned the following design principles for digital badging in the MOOC space.

Case 1 Design Principle: It takes more than a badge to motivate MOOC completion. The digital badges alone (as a microcredential replication of the verified certificate) are not a silver bullet, magic sticker, or shiny trophy that will extrinsically motivate more learners to complete a MOOC. However, offering digital badges could help identify early serious learners on a successful pathway and offer missing ingredients to the current credential.

Case 2 Design Principle: Balance high-quantity objective assessment with high-value social achievement. Designing badges at large scale requires a careful balance between high-quantity automated algorithm assessment (criteria = simply multiple choice quiz score 70 percent or greater) with a high-value criteria that includes peer-situated social value (All-Star and Coach badges showed distinction from peers). This also allows MOOC providers to target specific audiences.

Case 3 Design Principle: Integrate e-portfolios with digital badges to make learning performance visible. This allows evidence-based learning and brings the MOOC learners "alive" with a face and a story that shows proof of performance competency. In addition, the e-portfolio provides a logical space and place to hold the evidence and display the badge.

The MOOC study was an outgrowth of a Notre Dame badging initiative that began in 2012. Since the start 44 different badges have been created and issued to a total of 469 learners. We were recognized by the 2015 Campus Technology Innovator Award for pairing e-portfolios with badges to document informal learning on campus. Our initiative also includes an exploration of digital badges in professional conferences and educational development. In the future, we would like to take a deeper look at the model of badge engagement and the impact of visual design on badge effectiveness. We are also exploring the idea of integrating badges into existing credentialing systems.

Acknowledgments

This project would not have been possible without the research, design, and implementation collaboration from the Kaneb Center (Kevin Barry and Alison Lanski), Office of Digital Learning (Elliott Visconsi, Laurie Kirkner, Crystal DeJaegher, KC Frye, and Sonia Howell), the Office of Information Technologies (Ron Kraemer and Xiaojing Duan), and of course the MOOC instructors and TAs (Dan Myers, Anne Pillkington, Michael Hildreth, David Mayernik, and Jay Hobbs).

Notes

1. Dikran Kassabian, "The Value of MOOCs to Early Adopter Universities," EDUCAUSE Review, June 16, 2014.
2. Daniel T. Hickey, Nate Otto, Rebecca Itow, Katerina Schenke, Cathy Tran, and Christine Chow, "Badges Design Principles Documentation Project," [http://dpdproject.info/files/2014/05/DPD-interim-report-v4-january.pdf] Center for Research on Learning & Technology, Indiana University, 2014.
3. See the EDUCAUSE resource page on badging for more information.
4. Katy Jordan, "MOOC Completion Rates: the Data" (2015).
5. Jeffrey Young, "Online 'Micro-Master's' Programs Extend their Reach," Chronicle of Higher Education, September 20, 2016.

---

G. Alex Ambrose serves as professor of the Practice and associate director of ePortfolio Assessment in the Kaneb Center for Teaching and Learning at the University of Notre Dame.

Elizabeth Anthony serves as the Blended Learning Project Coordinator for the Alliance for Catholic Education at Notre Dame, where she wrote her senior thesis on digital badges.

G. Christopher Clark is an assistant director of the Kaneb Center for Teaching and Learning at the University of Notre Dame. He also serves on the faculty of the Wakonse Conference on College Teaching.