A large public research university figured out how to tap the power of artificial intelligence and human intelligence to produce impressive gains in student success.
Over the past ten years, the University of South Florida (USF) has experienced dramatic improvement in student success, as measured by the first-year persistence rate, the four-year graduation rate, and the six-year graduation rate. Each of those metrics has improved substantially. To promote persistence and completion, the university implemented a wide array of programs, practices, and policies based on a campus Student Success Task Force Report released in April 2010. These initiatives included many of the standard student success initiatives in place at many other colleges and universities, including living learning communities, the professionalization of academic advising, gradual increases in admissions requirements, course redesigns, expansion of on-campus housing, and promotion of student engagement.
However, from 2013 to 2015, in the middle of that ten-year period, retention and graduation rates leveled off. The First Time in College (FTIC) retention rate leveled off at 89% with the 2013 cohort and then declined to 88% (see figure 1).
Meanwhile the four-year graduation rate increased steadily until 2013–2014, when the improvements leveled off (see figure 2).
The six-year graduation rate hit the performance plateau at the same time, beginning in 2013 (see figure 3).
Organizing around Student Success
Since the state of Florida uses the FTIC retention rate and the six-year graduation rate to recognize and reward state universities for performance, USF placed the highest priority on increasing these two rates. As a result, they were compelled to find new ways to identify and prioritize students who were in need of assistance. They turned first to a homegrown predictive analytics model, saw some positive results, and then deployed a state-of-the art predictive analytics platform, from Civitas Learning, in 2014. When the tool first went live, administrators thought it would tell them what to do. They quickly learned that predictive analytics would do no such thing. Instead, they had to figure out how to put the analytical insights into the hands of trained experts who knew how to deliver the right support, to the right student, at the right time.
Next, in the fall of 2015, USF officials decided to consolidate three distinct organizational units into one. Student Affairs, an organization with 380 full-time employees and budget responsibility approaching $80 million, was merged with Undergraduate Studies, a complex operation with academic functions that include core curriculum management, academic advising, and new student orientation. These two units were joined by a third organization, Enrollment Planning and Management, which housed admissions, financial aid, and the registrar. The three entities became known as Student Success and are led by a single vice president. This arrangement gave all of the organizations a sense of focus and clarity of purpose. The goal was to advance student success, persistence, and graduation rates.
In addition, the intense, university-wide focus on these metrics led to the creation of a cross-functional Persistence Committee in January 2016. The combination of artificial intelligence, or constructed intelligence, with human intelligence and passion generated a powerful student success model that propelled USF off its performance "plateau" and is continuing to drive ever-higher rates of persistence and completion.1
Applications of Artificial Intelligence
USF employs artificial intelligence, or intelligence that is purposefully constructed, in various ways.
First Year Retention (FYR) Model
Researchers at USF developed a predictive model that has the purpose of identifying the chances that a student in the FTIC summer or fall starting cohort will persist to the end of the second fall semester. The FYR model uses pre-matriculation data and a logistic regression application to generate the persistence prediction. The data includes information from the admissions process (e.g., SAT/ACT score, high school average, gender, ethnicity, Florida residency). The second contributor to the model is from an administration of the Beginning College Survey of Student Involvement (BCSSE). The survey instrument is distributed and collected during summer orientation, and the return rate is very close to 100%. BCSSE data used for model development includes students' plans to work off-campus, academic and financial confidence, anticipated academic difficulty, and plans to study collaboratively. The model is generated every year based on the persistence experience of the most recent class. Sorting the FTIC cohort by the percentage chances of persistence, the 12% of the cohort with the least likelihood of persisting is identified for extra support, and those students become the cohort's FYR group.
The FYR initiative has been employed by USF since 2008, and the model has shown to have a reasonable degree of accuracy over the years. Predicting human behavior is almost necessarily inaccurate, but the model stratifies students by their need for support and identifies a group that the university can serve better.
Other USF-Developed Predictive Tools
USF officials became concerned about students whose grades were in the B/C range and who were not picked up by the FYR model. Many students whose first-semester grades were in that range were going to be fine, of course, but researchers wanted to better identify those who might benefit from extra support. Using the same data set as used in the FYR model (all pre-matriculation data), they developed a predictive tool to anticipate what each student's first fall semester GPA was going to be. If the model predicted a student would attain a 3.6 GPA, and the student instead received a 2.6, that indicates that a touchpoint might be beneficial, with someone visiting the student to go over performance, provide encouragement, and/or offer a referral for service if appropriate.
To address the four-year graduation rate, USF officials generated the annual project Finish in Four (FIF). It is an effort, in the summer of the student's fourth year, to encourage and enable graduation in August. That experience can be a frantic one for the students (and the faculty and staff who are involved), and much is determined in a short three-month period. To ease the pressure caused by FIF, researchers developed a tool that predicts students' chances of graduating in the fourth year (or before) based on records and data retrieved after the second fall semester. Instead of a three-month rush, this gives advisors, advocates, and the students five semesters in which to advance to a four-year graduation experience. The same model predicts which students, with support, could graduate in six years, another goal worth attaining.
Civitas Learning Predictive Analytics
The development and use of in-house predictive models laid the foundation and provided the impetus for deploying a larger predictive analytics model. As noted above, in 2014 the university contracted with Civitas Learning to deploy a predictive platform. This platform, drawing from the Student Information System (Banner) and the Learning Management System (Canvas), aggregates more than ten years of data and uses about 300 variables in a sophisticated model that predicts the likelihood that an individual student will persist into the next year (or semester). The platform allows administrators to sort students by college, cohort, race, ethnicity, gender, and much more. After selecting the desired student cohorts (which can include an entire entering class, for example), the Civitas model will generate (1) a list of the most powerful predictors of persistence for those students, and (2) a list of students sorted into five categories based on their likelihood of persistence.
Applications of Human Intelligence
USF and its students are highly dependent on planful, purposeful human engagement in response to predictive tools.
Stratification of Help Needed
The essential output of the predictive tools permits USF to get the right help to the right student at the right time and from the right person. University personnel recognize that many students are able to navigate their way through the institution using services and support as they wish. For example, the FYR model indicates that 88% of USF students will not need "extra" support or assistance, or if they do, they arrange for it themselves. The mean predicted persistence rate of those students is 93%. Their actual persistence rate, not coincidentally, is 93%. They do not need an extra "lift." Scientifically identifying students who do need support (the 12% who become the FYR group) is an essential element of the student success initiative. The other predictive models used by USF have the same effect: they stratify students' need for support. This allows university personnel to focus their attention where it is needed most.
Care Management
USF characterizes its student success initiatives as "care management." In some environments, this is called "case management," but the USF perspective is that support efforts need to be delivered in a caring way and that students need to feel as though they matter. There is a general expectation that everyone who serves students at USF need to do so in a helpful, supportive manner.
Academic Advocates
The Office of Academic Advocacy is staffed by fourteen "advocates," who work with students in academic distress. Many times that distress is driven by other matters affecting the well-being of a student, and the advocates are trained and oriented to refer students to resources that can help with those matters. Advocates triage concerns using a host of data points from various sources while cooperating with campus partners to generate the best support to students. Advocates are assigned to specific cohorts, so the size of the group they each serve is manageable. The work remains feasible by providing focused care to students using data-informed risk-stratification methods that are central to effective care management. The Office of Academic Advocacy began its operations in 2015 with three advocates. The growth in staff is a reflection of the quality of work they produce and is directly related to the improvement in student success.
Academic Advisors
USF has a largely decentralized system of academic advising, with each of the academic colleges employing its own team of primary-role academic advisors to guide students through major choices, course selection, and general academic decision making. As a resource for localized direct care to students, the academic advising community serves as an essential human resource in providing timely guidance, support, and intervention to students. The Council for Academic Advising is the coordinating body for advisors and serves as a mechanism to continually align the work within each college with the overall university student success priorities. Although advisors are selected based on their ability to demonstrate care and professional expertise, they are also engaged in technology-mediated advising practices using many of the tools mentioned above.
First Year Retention (FYR) Interveners
With ten years of experience applying the FYR predictive model, USF administrators have been able to refine the ways of connecting with those students identified as ones who might benefit from extra support. In the early years, students were called or otherwise contacted by administrators to establish a connection. That did not work. We learned that the best individuals for this contact were those with whom the student already had a relationship or with whom the development of a relationship was at least natural. Therefore, the FYR interveners are resident advisors (RAs) in the residence halls, instructors of Academic Foundations (the USF label for the freshman seminar), and academic advisors. RAs live with the students they serve, so when RAs are given the names of FYR-identified students, they can naturally make contact with the students and maintain a supportive relationship. Similarly, when Academic Foundations faculty are informed of students who would benefit from extra support, they can easily ask a student to engage in a conversation. And of course when academic advisors learn that a specific student would benefit from additional support, they can simply schedule extra time to spend with that student.
This may be one of the most effective uses of human intelligence in response to artificial intelligence at USF. People with whom the student already has, or will have, a supporting relationship are the ones who reach out to the student and form a connection. There is no structure to the intervention and no script for the conversation. The connection needs to be natural and "organic" so that it is comfortable for both parties.
Persistence Committee
In January 2016, USF formed a cross-functional team to manage student success initiatives, monitor where a student's success is jeopardized, and identify systemic and policy issues that interfere with student success. The small team of six or seven people is now a group of twenty-seven—an active team representing many corners of the USF landscape including Housing, Financial Aid, the Cashier, the Registrar, the Ombudsman, Academic Advising, the Office of Decision Support, the Dean of Students, International Student Services, and of course, the Office of Academic Advocacy.
Members of the Persistence Committee set up a workflow in which they receive weekly reports from the institutional research office, which generates the lists from the Civitas platform. Those lists "flag" any students whose behavior, as documented in Canvas or Banner, suggests that they are not likely to persist to the following fall semester. The Canvas data is almost real-time, demonstrating grades a student is receiving or whether or not the student is participating in online discussions and other documented learning experiences. Civitas reports are updated weekly to indicate if a particular student remains a concern or if the circumstances signaling concern have been moderated.
The committee meets on a biweekly basis to discuss ways in which student success can be enhanced. Members employ care management to choose the best USF resource to follow up with identified students. Advocates report on progress in key areas, and the committee reviews success rates by college, gender, ethnicity, and assorted other variables. The question of who is the best person to reach out to a student of concern is a central part of every conversation of the Persistence Committee.
"High Priority" Students
One classification of students has attracted the attention of the Persistence Committee and involved administrators. These are the ones labeled "High Priority": students who came to the university directly out of foster care and are wards of the state of Florida, who have experienced homelessness, or who have come from a family whose income is below the poverty level. Human intelligence and common sense, rather than artificial intelligence represented by data or a predicting model, generates this USF concern and motivates the university's support for those students. Caring people who keep the interests of students foremost on their minds are the driving force for USF improvement in student success.
Connecting Artificial and Human Intelligence
With multiple university partners invested in student success work using any number of data-supported methods, USF must intentionally design for alignment among all these efforts. For this reason, the Archivum Insights technology platform was created to facilitate improved communication and collaboration between student services areas using infrastructure tools: student search, notes, referrals, and cases. Starting in the fall of 2016, the IT organization built the early framework for Archivum, using the principles of care management and the early outreach efforts within the Office of Academic Advocacy as a guide. The first iteration of the system established notes (as a documented action) and referrals (as a call to action) as central mechanisms for driving passive and active cooperation among the student success professionals working to support undergraduate students' needs. Cases were launched to support the work of the Office of Academic Advocacy staff members as the student success network's "care managers" coordinating the information and resources provided from all partners, as a way to reconcile the confluence of academic barriers experienced by high-need student populations. All of this activity was stored and accessible within an Archivum student record strategically designed to display an overview of various student success factors (e.g., FYR, Civitas).
Additionally, the platform grew to include the development of (1) an assigned "Care Team" to display the diverse range of student success professionals providing direct support to students and (2) a graduation dashboard (e.g., "Finish in Four") to track essential data elements specifically identified as integral for improving graduation rates. The number of university partners who engage within Archivum Insights continues to grow—from early adopters within Academic Advocacy, Academic Advising, Financial Aid, and Career Services to others within Residential Education, New Student Connections, and Financial Education. In 2017, Archivum Insights was integrated with Canvas to provide faculty with direct access to the referral process without leaving the LMS environment. The following year, a robust appointment-scheduling dashboard was launched to provide students with online scheduling functionality and to connect new advisors' caseload management resources with the communication and collaboration framework within Archivum. The success of the system is demonstrated by the increasing numbers of referrals and appointments, signifying higher levels of engagement with the technology-facilitated collaboration.
Conclusion
The University of South Florida, a large public research university enrolling more than 50,000 students on three campuses, figured out how to tap the power of both artificial intelligence and human intelligence to produce impressive gains in student success. The two are inextricably entwined: one without the other would have left the university floundering. No matter how powerful or predictive an analytical model is, it still requires people—people who care about students and are trained to serve them—to provide the needed support and guidance. USF takes pride in its advancement of student success. Our reward? The students who benefit.
Note
- See also the USF case study: Paul Dosal, "Culture, Care, and Predictive Analytics at the University of South Florida," EDUCAUSE Review, December 9, 2019. ↩
Thomas E. Miller is Associate Professor, College of Education, at the University of South Florida.
Melissa Irvin is Assistant Dean of Undergraduate Studies at the University of South Florida.
© 2019 Thomas E. Miller and Melissa Irvin