From Search to Strategy: What Student AI Use Means for the Future of Academic Libraries

Gerry Bayne: So first, just generally, could you just tell us a little bit about your research project on student use of AI in the academic libraries? How did that come about and what was the impetus of the research?

Amy Deschenes: Yeah, absolutely. This actually started for us in 2023. We had an AI steering committee at the library, and all of our work at Harvard Library is we keep it user-centered as much as possible. So we do a lot of user research for a lot of different projects. And after the advent of ChatGPT at the end of 2022, it was like, this is really going to make a big impact on how people are getting their work done. We did an initial survey in 2023 with students about how they were using AI, when they were using it. It was very beginner. It was just the first step that was published in evidence-based library and information practice journals, so there's information there. What we talked about more at CNI was more recent work. So we did a follow-up survey as we've been hearing from students of what they're doing.

Amy Deschenes: And then also more recently, Meg led contextual inquiry and interviews with students to really just observe how they're using AI versus when they're using Google, Google Scholar, Wikipedia, traditional library catalog. Our library catalog is called Hollis. So it was really, really interesting. We were also taking a look at analytics at that same time because I think one of the big fears we kept hearing was are students going to stop using the libraries or are they going to just get everything they need from AI? And when we looked at the analytics for our library catalog, we weren't seeing that at all. It is consistent. We haven't seen any kind of drop-off. We do see some more traffic coming from AI tools, but there hasn't been any kind of significant drop-off. So what we are seeing is that they're making room for AI as part of their discovery process, especially for topics that are new to them, especially for things they're trying to get a sense of the field and they're really at that real, real basic beginner discovery.

Amy Deschenes: And then when they have more of a sense of either what they want to research or what they're looking for, that's when they come to the library for the credible verified information. So I think that is how we're thinking about discovery in the future. And so it's really being informed by the research that we did.

Gerry Bayne: So your research suggests that students are not abandoning. This just said not abandoning libraries as much as changing the order of the research process. AI comes first. As you said, I'm sort of repeating your answer back to you. So what does this shift mean for how academic libraries define their value? I know they've been struggling with that since almost the internet came out, so what's changed here at all, if you want to project? Yeah.

Amy Deschenes: Well, I mean, I think it just goes back to what I was starting to say was that we are a trusted, credible source of information for folks. That is one of the most important things about libraries is that you know you can trust the information that you find from a library. So I think it's even, as long as we are responding to the ways these search behaviors are changing, we're not expecting that people are starting with the library all the time. We've known that for a long time. People have been starting with Google, starting in other places, but then they come back to the library for that trusted information. I think the other thing we are thinking about is just how we can incorporate AI into discovery in some ways, but also understanding that people are going to use tools that we don't control. And I think another part of this is, and I know Meg has some things to say as well, just about AI literacy, making sure people understand how to check not just to automatically trust everything.

Amy Deschenes: And students are aware, students understand that there is this AI experience, hallucinations, all of that, and they have this more sensitive way of thinking about, am I going to trust this? Those kind of activities that they do to build trust with AI, but still it's never a hundred percent. They don't consider the chat a source in the way that you would consider a peer reviewed journal a source. And then I think the last thing is just the different ways we're seeing students using AI, sort of the high level executive use, which is quick answer. I'm in a lecture, they say a term I don't understand, I can look it up really fast and just get a super-fast answer. I don't need to go through search results. So that is where AI is providing a lot of value. But then there's that deeper, more instrumental use of AI where they're thinking of, you're almost using it as a thought partner, like spinning ideas up, brainstorming, that kind of stuff.

Amy Deschenes: And that's a more deeper engagement. And both of those ways of using it, there may be touchpoints that make sense for the library to play a role in those uses as well.

Gerry Bayne: I wonder if these results will change as time goes by, as AI gets more embedded and seeped into our academic culture. I've been talking to people about assessment and the idea of the game tape rather than tests. So I wonder if these things will shift, what you're finding will shift as AI becomes more of a partner and more engaged in every part of higher education. Do you have any thoughts about that?

Amy Deschenes: Well, I think it's also just the comfort level of people sharing how they're using AI. I think even since we first started talking with students and researchers and even just with our colleagues, at first it was kind of like, I used AI for this. It was really helpful. And now it is, I think, more accepted. So I think what you are speaking to also is as it is incorporated more into learning experiences in a formal way, they're hearing about it from their instructor, from their faculty member, from a trusted peer. It makes them feel like they can trust it a little bit more, but still there's something about the chat that makes, I don't know, people don't necessarily trust the chat outputs the way they would traditional search results. So that's just -

Gerry Bayne: That's a really interesting point. If you would put something in there and just get a list rather than saying, "Well, I'm glad you asked that or whatever," there's kind of a creepiness to the personality of AI, I guess.

Meg McMahon: Yeah, I will say from research that I've done on chatbot responses, students are more likely to trust outputs where it is a list where then they can go and continue on their own research journey on their own, opposed to seeing just things pulled from a resource and then having a tiny little citation under it. It's much better, at least from this research, which I think we did, 84 folks responded to it, so it was quite a few folks across and they appreciated things that helped them verify that that was a real source, like metadata, like a date, the authors. So when we're seeing and we're talking about credibility of these tools and will folks continue to go to a trusted source like a library, I think the answer is yes, because in the research when we're talking to these individuals, that's the question. They're like, "How do I check this?" They're asking to be able to do that in these spaces, even if it is something that the library's building that is essentially built on the resources that are trusted there.

Gerry Bayne: Yeah, that makes a lot of sense. Is there anything that surprised you about this research?

Meg McMahon: So I was just talking about a study that wasn't necessarily fully talked about at CNI. Are you talking about the chatbot response research or are you talking about the research of just overall?

Gerry Bayne: Well, I'll just read the question. You described students as thoughtful and skeptical users of AI, not passive users. What surprised you most about how students are evaluating AI-generated content, citations and research suggestions? You just talked about how they are doing that, but is there anything that surprised you about their behavior?

Meg McMahon: Yeah, so in the research where we were doing contextual interviews with students, it's definitely the sophistication of their skepticism. We found that students developed rigorous evaluation practices on their own opposed to someone suggesting to them, "Oh, this is how you evaluate AI."

Meg McMahon: A lot of these students were saying that I came up with this way because I wasn't getting information from my professors or from librarians on how I should be evaluating. So it's largely built based on their own experiences with these tools. So for example, we had one undergrad describe to us giving ChatGPT little tests, so, Asking it something that she knew the answer to related to the query or the area that she was interested in talking with it about, seeing if it agreed, and then calibrating her trust on that topic from there and deciding how she wanted to move forward with her research with the chatbot. So that's really verifying her understanding of the material with what the chatbot can give, right?

Gerry Bayne: Oh, go ahead. No, I was just agreeing. I mean, sometimes you get, from my perspective, because I don't talk to a lot of students, but I talk to a lot of IT and higher ed leaders, sometimes you get a little worried like, "Are the kids going to get it?" And hearing that they're doing this on their own and being rigorous on their own is very heartening. So anyway, I didn't mean to interrupt.

Meg McMahon:Yeah, no, no worries. I have more examples of that. We had a student tell us that they asked AI to explicitly pull up those external references to be able to go check it themselves rather than saying, "Here's the information," specifically prompting it to provide resources opposed to just an answer so they can actually go and learn on their own. So that's another way we talk to folks. And then another thing that I thought was really interesting was how students said that they believe their prior knowledge of the topic that they're interested in talking with the chatbot with actually shapes the quality of their evaluation. So they were cognizant of the fact that maybe I don't have a lot of prior knowledge on this, maybe I'm not actually going to be able to fully judge this. But the students who, when they were talking about chats where they knew they had more expertise, they felt better equipped to catch when an AI response is off, not just for hallucination, but also related to framing and actually if the response was useful for their thinking at all, because I think that's a really important part of how the students I talk to are thinking about these chatbots, not just accepting what it gives them, but it's like, is this actually even the answer to the question I'm trying to ask at this moment?

Meg McMahon: I had a grad student say it to me very plainly. She said, "The credibility of my research matters and I don't want to waste my time chasing a lead from a 1973 book that I know is garbage." So they're protecting their own scholarly standards.

Gerry Bayne: That's amazing. One of the strongest ideas in the presentation is that students are practicing research rigor, as you just said. Where do you see the biggest opportunity for libraries, faculty, and instructional teams to support students earlier in the process?

Meg McMahon: Yeah, so one thing I want to say is that we definitely heard from students that they want specific guidance on when and how to use AI responsibly. Like I was saying before, they're just kind of creating these practices on their own because they're not getting that guidance. And I wanted to mention a study that was done by the California State University system that surveyed over 90,000 people, and our conversations with our students map to that surveyed research where more than half of the students also want guidance on use there. So it's not just happening at Harvard, it's a thing that has been vetted outside as well, specifically help thinking through the ethics and limits of using these tools. So when we're thinking about curriculum and instruction, I really think we have to think of AI literacy not as a one-time workshop, but as a framework for judgment.

Meg McMahon: So the types of questions that that brings up is how do you evaluate a source an AI gave you? How do you recognize when AI has taken you as far as it can, and that there are limits within the line of questioning that you're going with it or the thing that you're trying to do? And then how do you protect your own intellectual voice when AI is a part of that process? And I think the library can be a part of this, but I also think it's scaffolding across academic departments at a university to help and make sure that these curriculum and skills are being literally built into the process at almost every point. If we're thinking specifically about libraries too, there's definitely an opportunity for a dedicated AI literacy role or an embedded librarian where they focus specifically on AI evaluation and helping meet students where they are in using these tools, not through a vilified model, but through a model of acceptance and being like, "Okay, this is where we started. Here's how we can move forward together as we think about your use of AI."

Gerry Bayne: That makes so much sense because I mean, I like that you're saying that it's on the user side because AI is changing so much. I mean, Claude now is not the same claud it was six months ago and it won't be the same claud it is six months from now. So keeping those guardrails on the user side makes so much sense. For CIOs, library leaders, and technology managers, what should this research change about technology investment? Should institutions be rethinking discovery tools, AI literacy programs, privacy policies, staffing, or all of the above? You sort of answered that in your last answer, so I didn't know if you had anything to add.

Amy Deschenes: Yeah, I mean, I can add a little bit here. I think that it's really not deciding whether we're going to respond to A or AI or not. It's here. It's how well are the systems that we have today and the services we have in place supporting the way people are actually working right now? The only constant in technology is change. It's always changing. And I think in libraries, we're always adapting, adjusting, evolving our discovery systems and the services we provide as user behavior evolves. So AI is another tool in people's toolbox, so we just have to think about how we are adapting to that. So I think it is all of the above. And at Harvard, one of the other projects we're working on is this AI-powered discovery tool for special collections called Collections Explorer, because we learned through actually our, I would say, previous large scale research project was focused on discovery of special collections.

Amy Deschenes: And what we learned from that is that our biggest opportunity at that moment was to make our distinctive collections easier for folks to discover. And so that is where we're thinking about our AI technology investment right now. So that's what I would encourage folks to do is say, "What problems do you have in your department at your university, whatever scale you're thinking about that AI could be a good solution for and a good fit for?" And if you're not sure, I would also encourage you to go talk to your users because I think that's the other thing. I think not only have we learned a lot that is useful for our library work in this, but I think just seeing how students are using these things impacted the way we are adopting them too, and just seeing they're three steps ahead of us most of the time.

Amy Deschenes: So as much as possible, keeping student and researcher voices in the mix when you're making decisions about resourcing and what are the real problems and opportunities that you have based on your user's needs.

Gerry Bayne: I want to thank you guys so much for your time and your expertise and for answering my naive questions because I'm not in the library world, so I don't know how a lot of this works, but it's very heartening to hear what you have to say and we look forward to hearing more. I'll link to any materials you have in the show notes so people can access this. Thank you very much.

Amy Deschenes: Yeah, thanks for having us. This was a great conversation.

Meg McMahon: Yeah, thank you.