Rating the Usefulness of Student Success Tools

Saying that student success is a priority in higher education is like preaching to the choir. Doubly emphasizing the importance of this issue, the EDUCAUSE 2020 Top 10 IT Issues list includes Student-Centric Higher Education (#5) and Student Retention and Completion (#6). Higher education institutions are focusing on how to become more student-centric and boost student retention and completion. But what does "student success" mean to higher education institutions and their students? Here's where members of the choir might be singing slightly different notes. Some colleges and universities define and measure student success by institutional-level outcomes such as retention and graduation. Focusing solely on high-level measures, however, may not provide institutions with "just-in-time" data on students' progress toward their individual educational goals. Instead, institutions need to identify the individual factors that contribute to students' ability to stay enrolled, succeed in their courses, and ultimately don their caps and gowns. If we don't, we will have considerable data on retention, persistence, and graduation rates but little understanding of what contributes to each of those measures.

Using metrics that can be collected sooner rather than later (i.e., leading indicators) is likely a good starting point to identify students' actions that lead to persistence and graduation. But what measurable leading indicators show that students are progressing toward these outcomes? Assessing students' engagement with online student success tools is a great place to start. Certain tools, such as early alert systems, are now part of the higher education landscape and can contribute to student success, particularly among at-risk students.¹ The EDUCAUSE Center for Analysis and Research (ECAR) asked students about their use of student success tools as part of its annual student technology survey.

But before we rush to measure the use of these tools, let's take a step back. The perceived usefulness of a technology is a major factor in whether it will be used. The perceived usefulness of a technology also influences students' attititudes toward technology. Identifying which tools students find useful (or not) may aid in assessing whether students are more likely to engage with available technologies that contribute to retention and graduation. If early alerts, for example, are not seen as useful, how likely is it that students will engage with them or follow up with academic support staff? Taking a look at what students and faculty find useful about these tools may offer some insight into whether the tools are valued and, ultimately, whether students and faculty will engage with them.

What Students Think

In our annual student technology survey, we asked students how frequently, if at all, they had used nine online success tools (see figure 1). Students who responded that they had used a tool were then asked to rate the usefulness of the tool they had used. The percentage of students who found these tools to be useful increased between 2017 and 2019 (see figure 1).² We also found that students' rating of online success tools varied based on the type of tool. Specifically, tools that help students with the practicalities of attending college—such as tracking credit hours, mapping out degrees, and auditing progress toward a degree—were found to be the most useful. However, students assessed tools that aided in their academic performance in individual courses (e.g., early alerts or tools that suggest ways to improve performance) to be comparatively less useful.

For each of these tools that aid in academic success, between 25% and 38% of students who were aware of their availability did not use them in the past year. However, certain groups of students find these tools more useful than other groups. For example, we learned in 2019 that non-white students found tools that improve academic performance to be more useful to them than white students did.³ Given that non-white students are at greater risk of not graduating, the fact that they see value in these technologies is promising.⁴

When it comes to measuring student success, gauging things like food and housing insecurity and mental health (in addition to good grades, persistence, and graduation rates) provides a more holistic view of the student experience.⁵ The inclusion of these more expansive measures is appropriate, given the increasing focus on identifying mental health issues among college students.⁶ So it's encouraging that students are giving more favorable ratings to things like self-service referral systems and social or community resources (e.g., volunteer opportunities, community events, and crisis counseling).⁷

Usefulness ratings can be linked with institutional analytics that measure students' use of online student success tools. Institutional leaders can identify, for example, which students are more or less likely to engage with particular technologies and can look for ways to increase engagement. Some of these institutional strategies may be broad institutional tactics, such as campus-wide communication campaigns that broadcast how these technologies work and what their value is (increasing learning and academic performance, for example). Such communication campaigns should be done in tandem with asking faculty to spread the word on how these tools work and why students should engage with them. If faculty find these tools helpful, and use them, they can reinforce with their students that engaging with these technologies can aid students in achieving their academic goals.

What Faculty Think

But faculty member's endorsement is dependent, of course, on whether faculty support these technologies. Again, faculty members' support of these technologies can be assessed by measuring usefulness ratings. Even at institutions where online student success tools are not available, faculty still see the value of the tools. For example, in 2019, a majority of all faculty we surveyed who did not have access to early alerts reported that if they were able to integrate that tool into their teaching, they would be more effective instructors.⁸ This finding holds true across all faculty types—from tenured to fixed term.

We asked faculty about four tools that are specific to their needs: early alerts, suggestions for academic resources to their students, suggestions on how students could improve performance, and tools that offer guidance about courses their students could take in the future. For each of these four tools, between 15% and 27% of faculty told us that they don't have access to them, and among those who have access to each of the four tools, between 33% and 54% of faculty said they don't use them. A majority of faculty who use these four tools find them to be at least moderately useful (see figure 2).

Given these faculty ratings, it may be easy to create buy-in for the integration of these success tools into faculty members' advising and teaching roles. On the other hand, if one-third or more of instructors aren't using certain tools when they are available, what does that tell us about how the usefulness of these tools is being communicated to their students? Encouragement for students to use these resources should come from all levels of the institution, not just from academic advisors or an app. Assessing faculty engagement with these tools is thus just as important as assessing student engagement with them.

And let's not forget the all-important influence of a student's social network. Research has found that if a student's peers find these tools useful, the student is likely to intend to use the technology.⁹ If a student hears from a friend that these tools can help in learning, or aid in tracking degree progress, the student may say, "I should look into using these."

Changing Perceptions and Increasing Usage

Understanding whether students or faculty find online student success tools beneficial is a practical entry point to measuring leading indicators of student success. Our research suggests that students and faculty generally find these success technologies valuable. Students deemed tools that assist them in navigating often-labyrinthine institutional systems (e.g., degree audit tools) as the most useful. Tools that aid in monitoring students' progress are rated as being the most useful to faculty. However, the tools that are most likely to measure leading indicators for persistence and graduation—that is, those that aid in academic success and learning—are found by students to be less useful. This also suggests that students may be less likely to take action after being alerted by these technologies. If these tools are not perceived as providing value, it is very likely that they will not be used or, at best, will be used grudgingly. Ensuring that these tools are pragmatic and worthwhile, and communicating that message to students, should be a priority for institutions. Working toward student success is important, and finding unique and meaningful ways to define and measure it is our starting point. And there can be more than one way to measure that success. Sometimes a choir can sing different notes, but as long as the notes are sung in harmony, the resulting melody will be amazing.

Notes

1. Hanover Research, "Early Alert Systems in Higher Education" (November 2014); John O'Brien, "Making the Case for Student Success Technology," Homepage (blog), EDUCAUSE Review, September 24, 2018. ↩
2. Dana C. Gierdowski, ECAR Study of Undergraduate Students and Information Technology, 2019, research report (Louisville, CO: ECAR, October 2019), 13. ↩
3. Ibid. ↩
4. "Status and Trends in the Education of Racial and Ethnic Groups," National Center for Education Statistics (website), page updated February 2019. ↩
5. Kathe Pelletier, "Student Success: 3 Big Questions," EDUCAUSE Review 54, no. 4 (Fall 2019). ↩
6. Gregg Henriques, "The College Student Mental Health Crisis (Update)," Theory of Knowledge (blog), Psychology Today, November 18, 2018. ↩
7. The usefulness of this online success tool was measured among students only for 2018 and 2019. ↩
8. Joseph Galanek and Dana C. Gierdowski, ECAR Study of Faculty and Information Technology, 2019, research report (Louisville, CO: ECAR, December 2019). ↩
9. Ronnie Cheung and Doug Vogel, "Predicting User Acceptance of Collaborative Technologies: An Extension of the Technology Acceptance Model for E-Learning," Computers & Education 63 (April 2013). ↩

---

Joseph D. Galanek is a Senior Researcher at EDUCAUSE.