Coaching Reports Support Customized Study Plans

In response to national goals of expanding access to higher education and increasing the number of graduates, many colleges and universities are working with students having more diverse educational backgrounds and abilities in self-directed learning than in the past. Western Governors University (WGU) has tackled this challenge since its inception, with its mission to expand access and offer competency-based degree programs.

Working successfully with a diverse set of students requires a diverse set of solutions. One solution is mass customization of study plans. However, customizing study plans for students at scale requires a technology- and analytics-driven solution. That's what WGU has implemented.

The WGU Approach

WGU is an online, competency-based institution with 32,000 students and 57 different degree and certificate programs. The university doesn't have traditional courses with required lectures and assignments — students earn credit toward their degree programs by demonstrating competence through direct assessment. The amount of time a student needs to study to pass an assessment depends on the student's prior knowledge, motivation, and ability. Helping students develop study plans customized to their needs is a major challenge.

Implementing the Coaching Report

WGU provides students and their faculty mentors an online coaching report based on each student's pre-assessment and assessment results by topic area. Topics are the core building blocks within the online curriculum of a single course of study, and each topic is linked to specific learning resources. These links are also available within the coaching report. A student can read the coaching report and click on the links to the learning resources in areas of weakness while ignoring the links in areas in which they have already demonstrated competence. This approach guides students to the most efficient path to mastery of a given assessment area. Figure 1 shows an example coaching report on 5 of the 16 topic areas in the course Fundamentals of Finance, Accounting, and Information Technology.

Figure 1. Sample coaching report

The student's faculty mentor uses the coaching report to follow up with the student. As one mentor explained:

"The WGU coaching report serves as a diagnostic tool for my student. As a mentor I am able to review the coaching report in detail with my student and direct them to areas of opportunity within the assessment. I'm able to assist my student with preparation for the exam by pinpointing areas that need additional development. And at the same time, I am able to build confidence with my student by pointing to the areas where my student is competent. The coaching report is a valuable tool to leverage strengths and areas of opportunity while streamlining student learning."

WGU allows students to reattempt a different version of the same assessment within the same term. In this way the assessments can be both formative and summative. However, students must contact their mentors to schedule each attempt on an assessment. The mentor can use the coaching report to guide the student and verify completion of further study of the area(s) of weakness identified from the previous attempt at either the assessment or pre-assessment.

The ability to provide this level of feedback to students through automation requires an explicit understanding of the mapping between assessment, learning objectives, and the curriculum. The coaching report takes advantage of the classification of learning objectives in courses of study at WGU. An average course of study is worth three to six competency units (roughly analogous to credit hours) and contains one or more assessments. Each assessment is organized into an average of three to eight competencies. A competency is further organized into specific learning objectives. Figure 2 shows an example of the link between competency, topics, and learning objectives, displaying one of the eight competencies in the Fundamentals of Finance, Accounting, and Information Technology course and the associated topics and learning objectives.

Figure 2. Mapping of competency, topic, and learning objectives for Fundamentals of Finance, Accounting, and Information Technology

Each learning objective is mapped to both specific items on the assessment and topics within the course of study. The learning objective is the key that maps specific resources in the curriculum to specific items on the assessment. All the relationships between assessment items, learning objectives, and course topics are managed by an application called the Competency Alignment Tool that WGU staff built. The Competency Alignment Tool has a web interface for the assessment department to enter data into a relational database.

Before the Competency Alignment Tool was built, WGU's assessment department had a well-defined system of classifying assessment items, learning objectives, and competencies. This existing system acted as the requirements for the Competency Alignment Tool. Since the requirements were well defined and the solution was simple, a single developer was able to complete the actual construction of the tool in one week.

Challenges

Defining and mapping the learning objectives at this level of granularity is the biggest challenge in implementing the coaching report. This labor-intensive process of careful thinking about the domain and student learning, though difficult, is extremely important. Once good learning objectives are written and mapped to topics, competencies, and assessment items, the technology and analysis behind the coaching report are simple. The technology is simply a series of web forms feeding a relational database schema named wguassess. The coaching report aggregates student assessment results by topic area, taking advantage of the assessment item to topic area mapping in wguassess. Because the coaching report is built on top of this relationship, its effectiveness depends on the quality of the learning objectives, the curriculum, and the assessment items.

Effectiveness of the Coaching Reports

The effectiveness of the coaching reports was assessed by comparing the mean pass rates on assessments before and after the implementation of the coaching report. Implementation of the coaching report took place in the six months between October 2008 and March 2009. Several populations were analyzed, using both simple mean comparisons and quasi-experimental controls.

First, pass rates for both groups (pre- and post-implementation) were compared for students who took advantage of the pre-assessment. The sample was filtered to contain only assessment/pre-assessment pairs that existed in both periods, yielding 116 assessment/pre-assessment pairs that existed in both periods, with 117,880 observations in the pre–coaching report period and 172,125 observations in the post–coaching report period. The average pass rate on assessments for this population was 72.1 percent before the coaching report implementation and 81.6 percent after the coaching report implementation — a difference of 9.5 percentage points (p-value < .001).

To control for differences in the assessments and students' prior knowledge between the two groups, a mixed logistic regression model was used. The assessment was treated as a random effect. Student scores on the pre-assessment and period (pre/post-implementation) were treated as fixed effects. Controlling for these variables, the marginal mean pass rate was estimated at 75.4 percent for the pre–coaching report period and 79.1 percent for the post–coaching report period. Controlling for these covariates reduced the estimated effect of the coaching report from 9 percentage points to 3.7 percentage points. The difference is still significant, with a p-value of < .001.

From May 1, 2011, to April 30, 2012, WGU administered 122,680 pre-assessments. Based on the estimated effect of 3.7 percent increase of mean pass rate, the coaching reports contributed to an additional 4,539 students passing their assessments, earning credit, and progressing in their studies. The coaching report is also shown to improve pass rates for students attempting an assessment more than once. The estimated effect on this population is a 6.6 percentage point improvement in pass rate. In the past 12 months at WGU assessments were attempted more than once 19,062 times, with the coaching report increasing the number of passes by 1,258. The total annual effect of the coaching report at WGU is thus 5,797 additional assessments passed (4,539 + 1,258). See figure 3.

Figure 3. Pass rates pre/post implementation of coaching reports

Replicating WGU's Success

The coaching report solution is applicable to any college or university that wants to provide customized feedback to students at scale. The technology and analytics are easily replicable. The Competency Alignment Tool can be built with on top of any database. It would even be possible to replicate it using Microsoft Access. The biggest challenge to replicating WGU's success is creating the taxonomy of learning objectives and mapping those objectives to specific learning resources. Though this process is labor intensive — even slightly painful — the benefits far exceed the ability to deliver automated feedback: Any institution that specifies learning objectives mapped to the curriculum and assessment items will gain greater insight into their learning objectives and identify areas of misalignment between learning objectives, curriculum, and assessment. Aligning those three elements benefits students and increases their likelihood of successfully completing their studies.