Discovering the Impact of Library Use and Student Performance

Key Takeaways

• Without a valid, reliable way to collect data from various library and enterprise systems, it's difficult to quantitatively assert how a library adds value.
• The University of Wollongong Library developed the Library Cube, a tailored database and reporting function that joins library usage data with student data, including demographic and academic performance information.
• Analysis of the resulting data reveals a strong correlation between students' grades and use of information resources the library provides.

By providing access to information resources, academic libraries play a significant role in the student experience. To date, efforts to assess the impact of accessing library-owned or subscribed content have largely focused on satisfaction surveys, feedback, and "return on investment" projects such as contingent valuation. Although surveys and feedback systems provide data and information on a range of service elements, they are limited in their capacity to provide information and insight into the perceived value gained by engaging with the library — that is, on a client's return on effort for using library services and resources. They are also unable to identify non-users effectively. Faculty awareness and knowledge of student use of library information resources is equally limited.

Collecting and Leveraging Reliable Data

At the University of Wollongong Library (UWL), we lacked ongoing valid and reliable data collection from both library and enterprise systems, which prevented us from making assertions about the value provided by the library. We needed a cost-effective, reliable, and sustainable way of collecting information on the library's impact on student outcomes. Our goal was to answer questions such as: What do students receive for their investment of time and energy when using library information resources? Does a correlation exist between usage of library resources and academic performance? What information can we extract from systems to enable faculty to better understand student success factors?

The University of Wollongong (UOW) uses a Performance Indicators Unit (PIU) to provide senior staff with accessible and integrated reporting and analysis solutions through a secure, web-based Performance Indicators Portal. PIU collects and leverages critical data through scorecard, reporting, and analysis capabilities, providing a one-stop data source for institutional data.

In 2009, UWL began working with PIU to develop a tailored database and reporting function — the Library Cube to join library usage data with students' demographic and academic performance data.

How the Library Cube operates:

Although PIU hosts a large historical data set for UOW student demographics and grades, UWL began collecting resource usage data for integration into the Library Cube only in mid-2009 for loans and in January 2010 for online resources. Consequently, the Library Cube only contains time series data starting from 2010.

Creating the Library Cube

The main requirement for joining any two data sets together is that each data set contain a common unique identifier. In our case, all of the systems required to create the Library Cube do contain a unique personal identifier — the student number.

The two UWL data sources fed into the Library Cube include loans data and data relating to usage of electronic resources. Due to the limitations of UWL's Library Information Management System, the only usable data that can be exported that includes the student unique identifier is the total number of items borrowed to date. To build time series data, an export of data occurs weekly, and the difference between two weeks is the borrowing activity that occurred each week.

UWL uses ezproxy logs (sourced from authentication) to extract information on usage of the library's electronic resources. Each time a user accesses certain library resources, an entry is generated in the log file. These resources include subscription databases, e-books, and e-readings materials, which can be accessed via the UWL catalogue or through links provided on UOW's learning management system.

Once exported, the data is contained within an Oracle Data Warehouse with access managed through the IBM Cognos 8.4 business intelligence toolkit. The entire system is developed and maintained by PIU. As  the Library Cube Video shows, users can simply drag and drop the dimensions of the Library Cube, such as gender, faculty, and country of origin, either as a field in a cross-tab table or as a filter value. Users can also select what they wish to measure, such as student head count or total number of borrowings. The web-based system lets users easily slice and dice data, as well as save their views.

Meeting the Challenges

The logs contain useful information, such as the students' unique identifiers, but they also contain a lot of extraneous data. Counting the number of log entries proved futile as they vary wildly depending upon the online resource being accessed. UWL decided to use the time stamp in the log as a de facto time session measure, using the following business rules:

• The day  is divided into 144 10-minute sessions
• If a student has an entry in the log during a 10-minute period, then 1/6 is added to the sum of that student's access for that session (or week, in the case of the Marketing Cube).
• Any further log entries during that student's 10-minute period are not counted.

Using this logic, UWL measures how long students spent using its electronic resources with a reasonable degree of accuracy due to small time periods (10 minutes) being measured.

The primary ethical and legal issue is privacy. The UOW's Privacy Information Sheet outlines the 12 principles to which the university must comply regarding the collection, storage, access, use, and disclosure of personal information. Fortunately, there are no legal barriers, as UOW has consent to use personal information for the project through its privacy consent framework, which students must agree to as part of their enrollment).

At an ethical level, the additional privacy risks potentially posed by the project have been eliminated by managing the personal information in a particular way. Privacy is an issue only to the extent that it involves the use and disclosure of personal information. UWL will not use the Cube to drill down to see a specific individual's personal information. That is, the data viewed in the Cube will always be aggregated. In all cases, the personally identifiable data that users can glean from the Cube is significantly less than that which can already be ethically and legally obtained through the library management system (LMS), logs, and access to student management systems. Moreover, access to the Cube will be even more restricted than is the case for the other systems that contain the same information.

Resource Use and Student Performance

Before looking at the relationship between UWL resource usage and student performance, it is important to note the following:

• Borrowing a book does not automatically translate into learning. The same logic applies to electronic resources.
• Many other factors besides the library contribute to students' academic success, not least of which are academic teaching skills and students' attitudes and aptitude.
• Correlation does not prove cause.
• Other variables that may contribute to students' academic success, such as attendance, either cannot or have not been captured in the Library Cube due to technical and resource limitations.

However, several factors increase our confidence in the validity of the findings, including:

• Sampling error is not a problem because the data is a census (updated weekly for loans and daily for electronic resources).
• There is little variability within the categories over time.
• There is a strong relationship between the average marks for each level of resource usage and student marks, and (with a few notable exceptions), this relationship holds for many views of the data.

Analyzing the data reveals a strong correlation between the use of UWL information resources and student grades. Figure 1 plots the average mark for each and every integer frequency of student usage of UWL's electronic resources. For example, the average mark for students who never used UWL electronic resources in 2011 was 55. The average mark for students who spent up to one hour a year accessing UWL electronic resources per year was 61. The chart shows a very strong nonlinear relationship between average usage of resources and average student marks (R-squared = 0.91).

Figure 1. Correlation between electronic resource usage and student grades

We defined the first frequency to contain less than 10 students as a cut-off point (to exclude the long tail). These outliers constitute 1.9 percent of the total (451 out of 24,141 students). To apply a logarithmic line of best fit, we incremented each of the frequencies by one. This has no impact on the correlation, the line of best fit's shape, or the points' relationship to each other. Also, the data excludes UOW offshore students and students from the Australian Command and Staff College.

As Figure 2 indicates, the principle of diminishing marginal returns apparently applies here, with the curve flattening with higher usage. It also appears that the strength of the relationship between electronic resource usage and students' academic performance weakens further along the curve.

In 2010, the correlation between borrowing and student marks was similar to electronic resources, but not quite as striking (R-squared = 0.73). In this case, the correlation was not as tight, and the increase in marks with usage was not as steep. In 2011, the correlation between borrowing and student marks dropped substantially (R-squared = 0.63).

Figure 2 and Table 1 illustrate the types of reports that both UWL and faculty can access and produce. The views outline the usage trends of undergraduate students from the Faculty of Commerce as well as the distribution of usage and impact in terms of the weighted average mark (WAM).

Figure 2. Undergraduate student resource usage (Faculty of Commerce)

Table 1. Weighted Average Marks (WAM) for Undergraduate Students (Faculty of Commerce)

The Library Cube is now part of the UOW enterprise reporting system and went live in May 2012. Representatives from UWL and PIU have met with key stakeholders within the faculties to communicate the purpose, scope, and functionality of its reports and views. Faculty members have welcomed access to another data set that helps them better understand the student experience and the impact on student success.

Professor Michael Zanko talks about the Library Cube:

Although the data illustrates the extent and impact of the Cube's use, it also poses many questions that will require further investigation, such as why some student groups are more successful than others when usage patterns are similar.

Key Factors and Related Efforts

Two factors have proven central to the Library Cube's success. First, without PIU it would have been impossible to create a sustainable system for linking library data with student data. Not only does PIU have the required technical expertise for completing what was a complex project, but they were also responsible for creating a data warehouse for student data — without which the Library Cube could not have been initiated.  It was also imperative that the UWL could access and use the ezproxy logs. There is a long established trend that UOW students are using electronic resources more heavily each year. Without the ezproxy logs the Library Cube would have only provided a fraction of the story about how the library is used and its impact on students' academic performance. Other projects have also attempted to establish correlations between library use and student performance.

• The JISC-funded Library Impact Data Project "aims to prove a statistically significant correlation between library usage and student attainment." The project uses activity data from three separate systems and matches them against student records held in a fourth system. "By identifying subject areas or courses which exhibit low usage of library resources, service improvements can be targeted. Those subject areas or courses which exhibit high usage of library resources can be used as models of good practice."
• The University of Minnesota Library Data and Student Success project aims "to show, using university metrics of success, what library use does for student success at the U of M. In addition, analyzing the connections between library use and academic success will provide evidence-based data that can inform service improvements and efficiencies, the development of new services, and the allocation of resources for necessary impact."
• The Association of College and Research Libraries Value of Academic Libraries Initiative is "a multiyear project designed to assist academic librarians in demonstrating library value."²

Conclusion

The ability to demonstrate the value and impact of libraries and their collections is becoming all the more important. Not only do we need to convince the university executives and faculty of the value of libraries, but our most challenging audience is increasingly the student body. We now have a compelling story to share based on the data generated to date: students who access library resources do outperform students who don't. The Library Cube, therefore, provides a new facet to better understand the total student experience.

The time required to establish the problem statement, business rules, and reporting requirements has been lengthy, but ultimately worthwhile. The UWL Cube, the JISC Library Data and Student Success, and the University of Minnesota projects all provide insight on the types of data sets that can be considered and collected, and identify key collaborators (e.g. PIU) and the business rules or algorithms that can be applied. Libraries interested in pursuing a similar project are encouraged to talk to their enterprise data centers or units to identify the potential for linking discrete data sets. Enquiries and requests for further information on the UWL Cube can be directed to the authors.