E2Coach: Tailoring Support for Students in Introductory STEM Courses

The University of Michigan is a large, public institution serving over 40,000 students every year. Many of the introductory courses are large lectures, with a professor addressing up to 550 students at a time. In these classes, professors advise the students as a whole about study techniques; they may also advise students one-on-one over e-mail or in office hours, but the size of the classes makes truly personalized communication between the students and the instructor — the kind that addresses a student's individual attitudes, goals, and background — impossible. Research in other fields, particularly public health,¹ demonstrates the efficacy of personalized communication in supporting healthy behaviors. By adopting technologies that deliver tailored text and data graphics to individual students, we seek to bring public health insights about motivation and behavior change to STEM education.

Over the past several years, with the support of the Gates Foundation–funded EDUCAUSE Next Generation Learning Challenge (NGLC) program, we have designed a holistic, software-based student-support system called E²Coach that advises students through their introductory STEM courses. E²Coach sends students personalized messages in the style of a written letter throughout the term. The messages also incorporate normative data graphics that encourage reflection and point them toward strategies for success. Because its messages are individually tailored, E²Coach can simultaneously support students who are struggling and push students who are thriving to extend themselves.

This adaptability is an important feature. We have something different to say to every student, and E²Coach lets us scale to any size class.

This article introduces E²Coach, describes its design and implementation, and discusses some lessons we have learned along the way.

How Does E²Coach Work?

Technically, E²Coach is a web application supported by powerful back-end software, called the Michigan Tailoring System (MTS), an open-source software package developed by the Center for Health Communications Research (CHCR) at the University of Michigan. The features of MTS allow us to address students by name, speak to their aspirations, and tailor communications to any information we know about them. The large number of tailoring variables combined with the student data — collected from the university registrar, surveys distributed to students within the E²Coach platform, and the departmental grade book — allow us to create a vast array of possible messages. This style of tailoring results in a message not for a subpopulation in a class, but instead content in effect written for a specific student. Figure 1 outlines the major components of E²Coach.

MTS has two main components: the Workbench and the Engine. The Workbench is a stand-alone desktop application that allows a writer to create tailored messages using select variables that are then sent as personalized messages to all users. The Workbench creates XML files that are read and rendered in HTML format by the Engine, a Python module that runs on the server. The messages are deployed through a suite of Django web applications, developed by the E²Coach team.

Following CHCR's example, two people work on each messaging project. The application programmer is responsible for the deployment of the system, handling tasks such as database and server maintenance as well as secure access for users. The message author writes the content sent to students. The Workbench is designed so that a message author does not need to have much computer programming background. Its GUI is structured as a grid in which a message is entered in one column, and the logic that determines whether or not a student will see that message is entered in another column. For instance, it can send the sentence "It's especially important for you to do well as you begin your college career because you'll use physics as an engineering major" only to students who are new to U of M and are engineering majors. A message author could tailor the message further by addressing the kind of engineering major, meaning a student might see the following instead: "It's especially important for you to do well as you begin your college career because you'll use physics as a biomedical engineering major."

Although no programming experience is necessary, working in MTS does take some acclimation. A host of workshops and documentation about how to tailor messages in MTS is available from CHCR's MTS website.

What Do the Students See?

To enroll in E²Coach, students fill out a detailed survey that collects basic information including their name and major, in addition to levels of preparation for the course, attitudes about science, study plans, desired grade, and level of confidence in that grade. Figure 2 shows a message sent to a student approaching the first exam of his course. Tailoring and the corresponding variable names are highlighted in color.

This particular message was designed to both offer encouragement and moral support, but it also aims to restore the student's sense of balance during a potentially vulnerable time by reminding him of his own values, which he wrote about in the initial start-of-term survey.

Since one data source is the course grade book, messages are also tailored based on the comparison between students' self-stated goals and their actual performance. Such messages can even be predictive, suggesting how an individual student might ultimately do in the class. These two parameters in particular allow us to advise students in a way not previously possible at scale. To a student who is on track to receive his expected grade, E²Coach's tone will be congratulatory. To a student who is not on track, E²Coach uses an encouraging tone and emphasizes the importance of changing study behavior to increase the chances of success. Similarly, verbatim advice from study group leaders and students who succeeded in the class is embedded in the system and matched to students based on basic normative information but also on the student's performance. While the tone of the message and the peer advice sent to students may change based on the distance between goals and reality, E²Coach suggests study tips to students at all levels. The system is designed to encourage self-reflection on study habits and the adoption of effective study techniques.

Graphics and Normative Information

Through the incorporation of interactive graphing tools — we use a library called HighCharts, which is free for noncommercial use — the system can send personalized graphics to students, answering key questions. For the common "How am I doing relative to everyone else?" question, E²Coach sends students exam distributions for their class. The system also generates a personalized grade-prediction graphic to help answer "What grade am I on track to get?" (We project what their final grade will be if they continue to receive the same grades that they have been.) We also send students historical data using data from previous semesters of E²Coach, showing distributions such as the number of hours per week past students planned to spend on their homework.

Initial Outcomes

E²Coach was first launched in January 2012 in four introductory physics courses and has been offered to three terms of physics students so far. Enrollment was voluntary, and 2,234 of the 5,523 students across the three terms chose to participate at a variety of levels.

Of those who enrolled, 898 used E²Coach infrequently (low users), 745 used it moderately (moderate users), and 591 interacted with the site frequently throughout the semester (high users). Usage groups were based on the number of visits to E²Coach pages (above or below the median) and the number of unique weeks in which students visited. Low users were on the low side of both measures (with two or fewer weeks of visits), high users on the high side of both (with five or more weeks of visits), and moderate users filled the gap between the two. To evaluate how E²Coach affected the performance of each student, a score was calculated that evaluates how each student performed relative to their incoming GPA, called the "better-than-expected" or BTE score. The higher the students' BTE scores, the better they did relative to their incoming performance.

We used a one-way between-subjects ANOVA to examine the effect of E²Coach use on BTE scores among non-, low, moderate, and high E²Coach users. Mean BTE scores differed significantly across user groups (F(3, 4845) = 11.07, p < .001), as shown in figure 3. There was a significant linear trend in mean BTE scores across user groups, in increasing order of program use from nonusers to high users, unweighted (F(1, 4845) = 26.17, p < .001). In other words, the more students used E²Coach, the more likely they were to have a high BTE score and thus outperform their incoming GPA.

Furthermore, students see specific benefits in using E²Coach, as the following current students express:

"During Physics 1 I did not use E²Coach, but for Physics 2 I have used it almost every day! So far, it has helped me to stay on track with my homework and to always be thinking about how I can improve the way that I study for physics and how to study more efficiently and effectively."

"I like the layout of what needs to get done every week as it keeps me organized. I like the messages that show my score as compared to the student body, as it helps to show if I'm keeping up with the pack."

"E²Coach gives great outlines for planning out my study sessions for each important topic. It gives me a good guideline and benchmark on where I need to be in order to do well on the test."

Building on the Past, Looking Toward the Future

We have learned some lessons by evaluating the E²Coach use data and looking for patterns of success in E²Coach students. We also collected feedback from the students. Our key lessons include:

• Avoid "survey exhaustion": During the first semester that E²Coach was live, we required that students take a survey to see their next message. While this further refined the tailoring as the semester went on, we saw high numbers of students who "dropped off" at these survey points and so did not receive further messaging. After the first semester, the only survey required was the initial survey, which provides the vast majority of the information needed.
• First impressions matter: Given the support that E²Coach can provide through the term, students ideally would visit periodically. However, within only a few moments of looking at our pages, some students found the system not helpful enough to be worth their time and decided they wouldn't use it. To further encourage engagement with the tool, we have gone through several design iterations, most recently adopting a sleek, powerful front-end framework, Bootstrap, which updates the look of E²Coach. It is also responsive to screen size, so E²Coach is now more easily accessed on mobile devices.
• Design at scale from the start: With so many moving parts in the system, it was tempting to create solutions specific to the moment. But for a system designed to operate at scale, these shortcuts created tedious work when changing semesters or when deploying E²Coach in different classes. Budget your time to implement scalable techniques early in the process.

While E²Coach was initially developed for physics at U of M, it is nonetheless a general-purpose design adaptable to almost any class. While keeping these lessons in mind, we recently deployed E²Coach in select introductory biology, chemistry, and statistics classes. We hope to further integrate it with other established campus resources, providing useful feedback about individual students for advisors and study group leaders to employ in helping students improve their performance.

1. Victor J. Strecher, Saul Shiffman, and Robert West, "Randomized Controlled Trial of a Web-based Computer-tailored Smoking Cessation Program as a Supplement to Nicotine Patch Therapy," Addiction, Vol. 100, No. 5 (May 2005), pp. 682–8; and Anke Oenema, Johannes Brug, Arie Dijkstra, Inge de Weerdt, and Hein de Vries, "Efficacy and Use of an Internet-delivered Computer-tailored Lifestyle Intervention, Targeting Saturated Fat Intake, Physical Activity and Smoking Cessation: A Randomized Controlled Trial," Annals of Behavioral Medicine, Vol. 35, No. 2 (April 2008), p. 125.