The Assessment Debate in an AI World

Sophie White: Hi everyone. In this episode of EDUCAUSE Shop Talk, we look at the new report on The Impact of AI on Learning Assessment and institutional perspectives on how learning assessment changes or has not changed in this age of AI with guests Liz Norrell and Hans van Oostrom. It was a really fascinating discussion to me where we started by defining learning assessment. It's a very hotly debated topic right now, how we look at really assessing learning, but also this conclusion that it's really difficult to actually measure learning. So how can we use the systems and processes and technologies that we have to do the best possible job to look at learning? And despite all of the hype around AI, we really came back to what are the foundational elements of learning and how can we use our human relationships to foster the best learning experience possible for students?

So we talked about some specifics, report data as well as ideas like grading and alternative grading. But I think that it was a really inspiring conversation to me and just how even though AI has upended some of how we look at assessing learning, it also is bringing together foundational approaches to making sure that we're truly doing the best job as educators to support our students. So enjoy.

***

Hello everyone and welcome to EDUCAUSE Shop Talk. I'm Sophie White. I'm a content marketing and program manager at EDUCAUSE and one of your hosts for today's discussion.

Jenay Robert: And I'm Jenay Robert. I'm a senior researcher at EDUCAUSE and I'm your co-host today.

Sophie White: Great. We are really excited to have a discussion today that accompanies an EDUCAUSE report on The Impact of AI on Learning Assessment. So we'll be chatting with some fantastic guests today about AI, learning assessment, higher ed, and all the things related to it. So I'll introduce them and then we will jump into it. First of all, we have Liz Norrell. Liz is the Assistant Director of Instructional Support at the University of Mississippi's Center for Excellence in Teaching and Learning. Her background is in journalism and political science, though she teaches across the liberal arts curriculum. Her work focuses on inclusive teaching, disability awareness, and pedagogical wellness. Thanks for being here, Liz.

Liz Norrell: It's great to be here.

Sophie White: Great. Our last discussion was about humanities in the age of AI and I studied English literature and political science, so I feel like we have a parallel track here in terms of our interests, which is fun. Next up we have Hans van Oostrom, who is the director of the AI Squared Center at the University of Florida, a provost level center, which oversees AI education across the curriculum. He's a faculty member of the Department of Engineering Education, a department he founded and chaired for six years. Thanks for being here, Hans.

Hans van Oostrom: You're welcome.

Sophie White: Great. Great. So we have a lot to talk about related to the report, learning assessment, AI, all of these things. But before we do that, Jenay, you were the lead author on this report and also have a lot of expertise in this. Can you tell us how you would define learning assessment in terms of the report and how we're talking about it for purposes of this conversation?

Jenay Robert: Yeah. And it's so important to start there because when you say the word assessment by itself, anyone could take it in any way. So we did define it upfront in the report for clarity for the reader. I'm going to read the definition. This was presented as well to all of the respondents from the survey and just a little disclaimer. The purpose is not to actually fully define learning assessment, which for any of my other teaching and learning nerds out there, you know that that could be a whole podcast in itself, but this was a starting point to say, let's all get on the same page about what we're talking about. So having done all of the hand waving associated, the definition that we used for this research was a systematic process of gathering, interpreting and using evidence to determine what students know, value and are able to do as a result of their educational experiences.

And this could be things that you traditionally think of as assessments like taking a test or a quiz, but it can also be things they could look a variety of ways depending on the discipline and the learning objectives and all of the things. And so it could be things like capstone projects or portfolio projects or skills demonstrations or papers or conversations. It could look any number of ways. The important thing for the context of this research and this conversation is that we're not talking about institutional assessment, for example, that your IR office might be undertaking. We're really talking about gauging what students have learned through their experiences at our institutions.

Sophie White: Great. That is super helpful. Anything you all want to add to learning assessment, how we're defining it, do you feel like that's an accurate depiction of how it looks at your institutions or is there anything you think we're missing?

Hans van Oostrom: Yeah, I think it's good to clarify it because we do institutional level assessment, we do course level assessment and we do assessment of student learning. And I think we should focus on student learning because that's really what is important and it's affected by AI and how we do it. So yeah, good definition.

Liz Norrell: I think the thing I want to add, and I think about this a lot, is that we can never truly have a perfect metric or measurement of learning because so much of this is invisible to even the learner. And so if the learner can't perfectly perceive how they've grown and changed what they know and what they can do, then it becomes even more difficult for somebody external to their cranium to do that. And so these are imperfect measures and I feel like I always want to foreground that in any conversation about assessment of learning.

Jenay Robert: Yeah, super important.

Sophie White: Yeah, that's such a great point. I feel like some of these stickier skills like critical thinking, it's so hard to define. Are we this good at critical thinking versus this good? And it will never be perfect, but we do the best we can. So I'm curious, the report that this is inspired by the introductory sentence is few areas of higher education have been as passionately debated as learning assessment in the age of AI. So it's a provocative topic and I'm curious, why do you all think that this specific topic about how we measure learning in this age of AI is so fraught with issues and challenges and there are a lot of heated feelings about it. What are you seeing at your institutions and why do you think that's the case?

Hans van Oostrom: Yeah, I think it is true, right? So it's a very controversial issue, but I think we have to break it down to what are the fundamentals? What is it really that we're trying to achieve and how we're achieving it? I am not convinced that AI is really upsetting everything. It is exposing issues with assessment more than it is turning it upside down. I think I've said this to others before. I think we've become a little lazy with assessment since COVID. I think there's really a pandemic influence here when everybody went online and the assessments were somewhat watered down, maybe made easier for faculty to administer with an online quiz that self-graded. I'm not saying everybody's doing that, but I think we can't forget that that happened because a lot of people haven't come back to where they were before. And I think authentic assessment and really we got to go back even foundationally to look at what is it that we want to assess?

What is it really? I mean, we just said it. What are the important parts? What is this course trying to convey? We're not conveying memorization of knowledge. We're really conveying a higher level thinking on the material. And so it is difficult to assess, absolutely difficult to assess and not an easy thing, generate a quiz and click a button. But I think we've watered down our assessments and now AI comes along and AI exposes that. AI says AI could do this very easily because of the level of assessment we're doing. And so I think we have to really carefully think about that and rethink it. So yes, we have to change how we assess, but frankly, that was long overdue.

Liz Norrell: I have a lot of thoughts of Hans, and I can tell you and I are going to have a really fun conversation today, which is great. The first thing I'll say is I believe that in this moment AI and its role in the assessment of learning is probably the most controversial and heated debate, but I'm not prepared to say that it is the most divisive debate that we've ever had about assessment of learning in higher ed because I think we can go back to the beginning of the study of learning and see every time there's a new technology, there is an accompanying moral panic about it. And so I think this one feels really fraught because it's happening right now and the world seems very uncertain, especially because of the long tail of the COVID pandemic. With respect to whether assessments got watered down during COVID, maybe they did.

I don't know. What I do know is that all of us, students, faculty, staff, administrators, the general public just had less cognitive bandwidth to work with during the onset of COVID. And so maybe assessments got watered down or maybe we were just meeting people where they were. And I don't want to use words that suggest a value judgment on that, but what I do know is that the way that education was working prior to COVID simply wasn't working for a lot of students in a way that became very visible in March of 2020. I remember I was teaching in an in-person class when we moved online, someone who had never spoken in class before was suddenly one of the most active participants in our synchronous Zoom classes because some of the barriers had been reduced. And so I don't want to suggest that everything about pandemic learning has made things easier or less rigorous or less challenging for students.

I think it's just different. And so I certainly, certainly agree with the premise that this is a hotly contested area of discussion. And I think that's probably why Hans and I are here because we have different perspectives on that.

Hans van Oostrom: No, I think you're totally right. I think there was a whole range of ways people went online and did their work. I mean, people were more experienced than other people and some quality was higher than others. I just feel like we have almost not moved back to where we were before and I'm sensitive to what you're saying too. I think the whole learning process is a personal process and so each student experiences that differently and we talk about that all the time. You got to have different ways of teaching the material so it sticks with one and then it sticks with another. So I get all that and I think assessments is in a way that way too. People have test anxiety, you give them an oral examination and they're fantastic, but we don't have time to do oral examinations. And so I do think it's a personalized quest.

I do think that with AI, we can continue to do what we've been doing. We have to really carefully really go back to the roots of what is it that we want to assess. And I've been out of school a long time and if I look back at my career and what I used, I studied electrical engineering. I don't use any of that. And I think what really sticks is the critical thinking skills, the problem solving skills, the ability to provide leadership. There are so many things that people are needing to do in real life that I can't look back and say, where did I learn problem solving? Well, they gave me a lot of problems and I solved them, but was that a really structural way for me to learn problem solving? No. And so I think some of these things we have to start thinking about too, because there's a whole question about what's the value of a college education.

And to me, the value is that you learn to critically think, you learn to problem solve, you have a domain that you have a base knowledge on, but that base knowledge isn't going to get you through life. You need to actually keep learning lifelong learning, especially with AI. I've been doing some AI for 30 years, but it's changed completely in the last six months. And so if you're not able to continuously learn, I think you're going to be in trouble. And so how do we encourage that and how do we assess that? And I think if you do that, if you have a way to figure out, can somebody critically analyze something and think AI isn't going to easily be able to do that. That's a true human skill that we have to learn. So I think back to the roots, we have to go a little bit back to the roots.

Jenay Robert: Yeah. It's funny to both of your points here about going back to what has assessment looked like for a long time and kind of challenging those norms. When I wrote this report, we go through all these steps at EDUCAUSE and one step is editorial review and I want to read a couple things. So we have a section in the report about promising practices. Where do we go from here? What did respondents suggest as next steps for people? A couple of things stood out. Be clear about your assessment goals and use authentic learning assessments. And then there's some description of what that is. And another piece, scaffold learning activities and use a variety of assessment types along the way. And in these two items in particular, it's kind of funny because when you read this, AI doesn't show up anywhere in the description, right? And our editor, as he was editing the report, he said, "Well, couldn't this be true for any assessment advice?" I said, "Yes, that's the whole point.

That is the whole point." And I think gets right to the heart of why this is such a hot topic. As Liz as you pointed out, and it has been for a long time. For me as somebody who's worked in higher education for quite a while, the exciting part to me is that the introduction of this technology is pushing people to really... Now you have to evolve assessment practices. We've been talking about it for a long time and now it's kind of just right in everyone's faces.

Liz Norrell: Yeah. I think a lot of us have. . . I'm going to use this phrase and I don't mean this in a derogatory way, kind of gotten by with replicating the teaching methods that we experienced ourselves as students and that was good enough to do what we need to do so that we can do our research if you're in an institutional context where that's important. I have been saying I think since 2021 that these tools are an invitation, maybe even a requirement for us to reexamine business as usual and to think about perhaps good enough wasn't actually good enough and we were leaving people behind, but we didn't see it because we didn't have a tool that made it easy to fake it.

Hans van Oostrom: Yeah. I think another piece that has come to light too in assessment is the proctored versus unproctored. And I think that's a really important aspect too because I did an analysis. I'm the chair of the University Curriculum Committee, which is overseas all curriculum at the university. And I grabbed the last meeting for the catalog, which was March, I think, and there's a lot of items on there. There were 25 courses I think on there. And I looked at their assessments and tried to determine whether they were proctored or unproctored. And I found that over 50 percent of assessments in those courses were unproctored and there's nothing wrong with unproctored work, but you have to take it at that value that you are not guaranteed that the student did the work, you're not guaranteed that AI didn't do the work. And so you may want to think about the value of that assessment towards the final grade because you may not for sure know that you're assessing the student.

Assessments are not only to evaluate the student and where they're at, it's also to give feedback to the student to figure out where they have to focus their learning in the class and that's an ongoing thing. And so some of these unproctored assignments can do that if the student chooses to do the work themselves. So there's room for that, but I think we have to also carefully think about how do we then ensure that we're assessing the student in the end? And that's also the question with AI, but the question isn't truly new. We have other mechanisms. There are services online you can send your assignment to and somebody in India will prepare a wonderful report for you. Now we have AI, maybe that's even easier, but it is not truly new. And again, we have to think about those things too is how are we assessing, what is the assessment for?

Is it for the student to get feedback to see where they're at so there's maybe a more higher level or a midterm on an exam or some other culminating examination and a lot of people don't do those anymore, but say a final project and I think we don't necessarily think about it. I think Liz is right, "Oh, it was good enough before, so it's good enough now." And that's really, I don't think it's true.

Sophie White: So I went camping this weekend. This is a related story and it was a family trip and I was talking to an uncle who he's in his 60s, has been out of school for a while and he was telling me about CliffsNotes and how pre-internet you used to have to send away for CliffsNotes ahead of time to show up in the mail in order to help you cheat on whatever assignment you're trying to do. And that was so fascinating to me thinking about assessment in this age of AI because it sounds like if our goal as educators is to catch students doing something they shouldn't be, there's always a way to do that. But if our purpose is actually to help students learn, that's a completely different conversation altogether. So I just thought that was really interesting. And I know Jenay, we've been hearing a lot about process oriented learning, that's some professional development we're seeing at EDUCAUSE too.

So it feels to me like this is again, bringing up maybe some gaps that already existed but are coming to the forefront in this age where AI is so prevalent.

Liz Norrell: Sophie, are you saying that you don't remember CliffNotes like this was in piece of news to you?

Sophie White: I had the internet version if I wanted to. So this is not news, but I think that actually sitting away to them. Sorry.

Jenay Robert: They came in these cool yellow books. Yeah.

Sophie White: That's so funny.

Liz Norrell: Thank you for giving me my age reality check again to better. Yeah. Hadn't had one yet.

Jenay Robert: I'm sorry you missed out Sophie on the paper version because they were small, they were brightly yellow colored, like you always knew when it was a CliffsNotes book and often passed around so much that the pages were falling apart.

Sophie White: That's great. Well, I-

Hans van Oostrom: I grew-

Sophie White: Go ahead.

Jenay Robert: Go ahead, Hans.

Hans van Oostrom: Well, I was just going to say, I grew up in the Netherlands and our examinations were a final exam at the end of the course. I was the only examination that determined your grade and it's a totally different system. They don't do that anymore because it's terrible. I failed many classes because I didn't prepare well enough for the exam because I was supposed to do that the whole semester, but I didn't. And so there are different regions of the world doing things differently too and things evolve. I think if I look at what I did, yeah, no, that's easy. You write one exam and you grade it and you're done. That's wonderful from a professor's perspective perhaps. And if you have a large research portfolio, you don't want to spend time on teaching, but that's no longer accepted. So I think it is important to really help the students.

That's what we're here for. We're here to teach the students and to make sure they learn and we're not here to punish them for things that they're doing. We should be thinking about why they're doing this. Why are they not engaged in the class and learning it themselves? Do they not see what the use of it is or how it affects their future? I think we got to drive the students to motivate them and then we shouldn't have a problem with cheating because ultimately if you really want to love to learn, I think that's very difficult and we got 61,000 students, they don't all love to learn and they're here because whatever the reason is, right? It isn't necessarily their own choice to be here, but I think we have to show how it's relevant too. And I think I've built a lot of curriculum over the years and I think it's so important not to just stack knowledge upon knowledge upon knowledge for the sake of knowledge.

I think you want to show how is this preparing me for this job or that job or a career in something? And we don't always do that well enough.

Liz Norrell: I just want to jump in because I feel like whenever we have these conversations, it's important for us as I've said similar things earlier in our conversation here that I want to come back to the, I don't know, can we call it a fact, but certainly there's a lot of evidence to suggest that the single best intervention that we can do to promote student learning is to create smaller classes and that has nothing to do with technology. It has nothing to do with proctoring versus not proctoring. It has to do with the relationship on which we base the learning and Hans, you said earlier, this relational, this human enterprise of learning happens in community. And so we can chase after the AI tools all day to try to do learning at scale, but the most effective thing we could do is create smaller class sizes so that instructors can, as you were saying, Hans, do oral exams if they want to because it's not 400 people in a class that they would need to do that with.

And so what we're talking about here is tools that we're using that are not created for the purpose of helping motivate students to do really cognitively challenging work. So we're taking tools that were not designed for this purpose and we're trying to figure out how to use them in ways that might maybe yield the results we want and to catch people who might be using them in ways that we might not want them to do, but we're forgetting that the most important thing is that relationship basis for learning.

Jenay Robert: Yeah. And I'll add too, I think you're both spot on in terms of the motivation and the sort of the systemic issues as well. And so to me, in addition to those items, I often think about what are the barriers we need to be removing that perhaps we're not always cognizant of or that we don't have control over. How do we then mitigate the negative impacts of those barriers? So I mean, I sometimes talk about this. I share the fact that I'm a first gen college student, I didn't know how to do college, I didn't understand the system. I did not have time to study. I was working three jobs. I was the primary caretaker for a parent and a young sibling. So going through school was incredibly challenging for reasons that had nothing to do with my motivation or my drive or my will to succeed.

And that's actually witnessing how the system of higher ed can often be the biggest barrier to success, both through my personal experiences and other students is what brought me to this career in the first place. And so I just want to make sure that that's part of the conversation as we think about what would drive a student to use AI tools for cheating, which by the way, if you look at the research, it's not as big of an issue as everyone thinks it is, but let's just address it. It's the elephant in the room. So what would drive somebody to cheat for any reason? And there are just so many reasons that don't have anything to do with a student not wanting to learn or not wanting to be present for the experience.

Hans van Oostrom: Yeah. I think there are a number of, and you mentioned it in the report too, there's a number of categories of that. So I think clarity of what tools you can use and what tools you cannot use is important and we're just slowly graduating to make that clear. I think a lot of people basically say, "Well, I don't have to explain plagiarism. You're not allowed to do plagiarism. So why should I explain that you cannot use AI?" And I think you do have to be clear with all of it. I appreciate you mentioning your first generation graduate and I am as well. And you too, Liz, okay, yeah, you picked a good group. But I think that while that hindered me clearly just in the same way you mentioned, I didn't know anything about college and I had nobody that helped me. And so I almost failed my first year and was out.

And now I'm at a great university getting the opportunity to do all these fun things. And I think it helps drive you too a little bit. So then you always work harder to try to get there.

But I think we do need to realize that there's a whole range of students out there and we need to try to cater to all of them. And as your report said, I mean clarity of work, what can you do or what can't you do is one thing. I think the other part is a lot of students cheat when it's easy to do. So I have a roommate who's taken this class before. It's easy for me to let the roommate do some work or maybe I buy them a beer and they can do a little bit of my work.

And that is tied to what's the complexity of the assignment? Is it something that AI can easily do or not do? Is there proctoring? Is there no proctoring? And our Dean of Students' office is overflowing with cheating cases related to AI. And I've talked to our ombuds It's about it. And what are those cases? Well, they're all in these no AI courses where you shall not absolutely not use AI, but they haven't changed their assignments for decades and AI can easily do them. So yes, we should encourage students not-- to do their own work even if it's... But it's oftentimes they don't see the relevance of it. And I think with AI and some of the tools, I mean some of the relevance does go away.

I'm in engineering, we don't do drafting on a big table with pencil and rulers anymore because we have tools that can do that. And in the beginning people said, well, that's a skill they have to learn because of the spatial visual. And sure some of that is true, but at the same time the world has moved on. And with AI that's a little bit like that too. Let's figure out how we motivate students and give them assignments that motivate them to do it themselves, create things that are relevant that students are actually interested in doing and learning at the same time. Yeah.

Jenay Robert: My undergrad degree is in chemistry and I remember when I started the American Chemical Society had just dropped the requirement for accreditation that an undergraduate chemistry program had to include proficiency in German and I think one other language, I can't remember, because all of the seminal works in chemistry were in these specific languages. But at that point the technology was catching up to say, this is all now translated so you don't have to learn these things anymore. But yeah, I mean, I think that's something that in a discipline specific way, we're still trying to figure that out as AI technologies evolve, what are those skills that people need to still maintain as strictly human skills and what are those things that now there's new technologies that are going to help us get some of these things done?

Sophie White: Jenay, I'm curious, just to make sure that we have time to talk about the report, were there any elements of it that stood out to you as maybe surprising or particularly interesting that we should make sure to talk about?

Jenay Robert: Yeah. You know what? I was actually surprised I pulled up the statistic before this recording because I want to make sure that I was quoting it. So 92 percent of respondents agreed that when used correctly, AI tools can support the purpose of higher education and just 7 percent of respondents said that AI undermines higher education. And that to me was really surprising. I knew that there would be a fair amount of people who said that they think they can use AI tools to support the purpose of higher ed, but I didn't think it would be like a landslide. Now, who is our sample? This is EDUCAUSE. People who are taking the survey are probably more on the techno-positive side than others and there are all of those caveats that you have to keep in mind. Even so at EDUCAUSE when we give surveys, we definitely have critics of technology in our community.

And so to see such a high number, even for an Educause survey was really surprising to me. And then similarly surprising 92% saying that higher education students should learn to use AI tools. And so I think that this probably comes from the idea that we're not just talking about generative AI chatbots, which is often what these conversations get reduced down to. Should everyone know how to have a conversation with ChatGPT and prompt the tool correctly? And I couldn't care less. I see Hans shaking his head.

I couldn't care less if every student is able to carry on a conversation with a chatbot. But I do imagine that for most of us, we're engaging with AI tools of some variety in some context, whether it's in our professional life or our personal life and we should reasonably know what the appropriate way is to either engage or choose not to engage with those tools. So I think that the data makes sense. I just didn't quite anticipate that level of agreement.

Hans van Oostrom: Yeah. I read that too, and I thought that was interesting because I did not think about that either. And I thought the same thing. Oh, maybe the audience is a certain way, but still that's huge numbers.

Liz Norrell: And it struck me as well, but what I just want to highlight is that that phrase at the beginning of it, when used correctly, is doing all the work. So where we disagree is what is a correct use of AI, I think, not the... Because for some people, I am fairly skeptical. For me, the correct use cases are pretty small. There are a few of them, but I would agree with that statement.

Jenay Robert: Right. Yeah.

Hans van Oostrom: I think it comes down to education again too, because when you said prompt engineering is that learning AI. And I agree. No, it's not. That's just how to operate a tool that is an AI tool. So we're trying to implement AI across our curriculum in all our 16 colleges and we take the approach that there's sort of three pieces to that. One is fundamentals of AI. How's AI work? Because if you don't understand what goes into these models, how can you understand what comes out of it? You don't know how these models are built, what they're all based on and you need some basic understanding. You don't need to understand the math, but you do need to understand what a model is and how it works, how it comes up with things. I think the second part is ethical and appropriate use and it's tied to understanding how these models work.

How do you now use it? What things should you watch out for? Are you going to base your decisions on a model that was slighted in one way and you're trying to use it some other way? And we have to come to a realization too that these models that are out there are using every single bit of digital data that's out there. Do we trust all that data? I don't. I'm on the internet every day. I see things that don't make sense, yet these models are using that as truth as data and it's going to come out at some point when we're talking about hallucinations and some of that is related to that. So I think we need to make our models better with high quality, validated data. We're not there. And then the third part of our training is application to the discipline. Now you've learned the foundations, how can you apply it in your discipline appropriately?

And I think that's really the approach. Prompt engineering is just... I see a lot of schools that buy a ChatGPT license for all the students and that's it. Now they're going to go out and teach themselves. It's terrible. How are they going to learn to appropriately use it? They won't. You need to guide them. You need to teach them. That's like giving them a textbook and say, read it and I'll give you a test. That's not what education is. And so it's a struggle though. People want to do the shortcut of, oh, I'm really good at using AI because I'm really good at prompting, but you really need to understand more. Again, education is power, right?

Liz Norrell: I'm just always so struck by how all of these conversations about AI always end up back at basic principles of what is education, what is learning? How do we motivate students? How do we create critical thinking? And AI is just one in a long line of challenges to those fundamental questions that are our work.

Hans van Oostrom: Yeah--

Jenay Robert: We're saying this in 2020 too, right? Back in 2020 or 2021, with the pandemic, when we were having so many issues with how do we do online learning in an emergency way, that was a similar situation where we were just shining light on all of these challenges we had already. And it wasn't just about how people learn or how to assess what people learn. It was also about what's fair and equitable in terms of access to education. Liz, you mentioned having some students who did so much better once they moved online and some students really struggled so much worse when they went online. And so it was less about that issue, that core issue that we were all trying to argue about. And it was much more about that foundational, how do we serve all of our students?

Hans van Oostrom: Yeah. And I like steering it back to these foundations. So whenever we have these conversations, I say, "Well, what are you trying to do and why are you doing it?" Because that already diffuses a lot of this discussion. Well, I do it because I've always done it. I do it because in our curriculum we've been doing this for 50 years. Well, maybe the world has moved a little in fifty years. So I mean, it is a good way to direct people to these foundational things that really need to be re-thought every so often and we have a great reason for it now, but I don't see a lot of effort happening there. So that's, I don't know how we motivate people to actually do that because that's difficult. I mean, we have 6,700 faculty here at the University of Florida. I don't see everybody jumping on that and say, "Oh, I'm going to change everything I do."

There may be a few, but...

Sophie White: Yeah. It seems like so much comes down to those relationships and communication between groups. In this report, one thing I was struck by, and Jenay, maybe you can speak with this, is that there were differences in responses between executive leaders and the rest of the respondents. And the takeaway was executive leaders were more likely than faculty managers and frontline staff to report that they believe students at their institution are enthusiastic about AI and that they have clear policies. And it sounds like the folks who are maybe working day-to-day with the students do not feel the same way. So to me, this says we maybe think we have communication figured out, but there are a lot of holes that need to be continually nurtured, relationships that need to be cultured as we figure out this moment together.

Jenay Robert: Yeah, definitely some more, I guess you could say, optimism on the part of some of the executive leaders. And I want to say in fairness to the executive leaders, these weren't enormous differences. It's not like all executive leaders were super optimistic and everybody else was super pessimistic. It was just enough of a difference to see that there might be an opportunity here for our executive leadership to look at that local data. At least I do when I write an EDUCAUSE report, I always encourage the reader to understand exactly how generalizable is this data. It doesn't mean it maps directly onto what's happening at your institution. So we always provide the instrument alongside the report and we say, "Look, give the same. If you think some of these things could be happening at your institution, give the same survey at your institution. I've written the survey, you can take it and see what's happening at your institution." And that's so important, especially as we're talking about certain things like are our policies and guidelines being applied consistently across the institution or do we have the right balance of institution level guidance and individual instructor flexibility, which is another really important thing that came out of this report that our respondent said, "Yes, I want guidelines from the institution, but I also want to make sure I have the flexibility and the autonomy to do what's best for me and my students." And so yeah, I'm glad you brought that out, Sophie.

It's a really important finding.

Hans van Oostrom: Yeah. And I think I'm one of those administrators and I'm positive, but at the same time, I try to understand all the aspects too, which is often difficult because I don't know all the disciplines. I don't know all the issues related to that. I think from a policy perspective, I think we have to be very careful not to create policies that conflict and overlap with existing policies. We already have a lot of policies. I've been a faculty member a long time and I was a department chair and I know policies, nobody loves policies. And so to have clear policies that are easy and reasonable and it can be implemented, I think it's important. For AI, we're a state school, our government, our board of governors wanted an AI policy and we're like, we don't really want to write one. We have policies that data security policies, then honor code, we have a lot of things already that yes, we can make clear that AI is a part of that.

And so we did write our policy, which is really an index to other policies and sort of said AI is part of all of this too. And I think oftentimes people want more. I want to see a policy where it says, "I cannot do this." So if you come to that point, I think you failed in trying to encourage somebody to do the right thing.

And yeah, I mean, we sometimes treat these things as legal things because people are going to sue and that stifles everything in the end.

Sophie White: Yeah. These are great points. And I was thinking about this, tell me if this is a terrible topic to discuss, but I was thinking about policies and learning assessment this week as I was reading about the Harvard decision to limit the amount of A's that could be given out in a specific course. So from what I was reading, it looked like faculty were overwhelmingly voting for it, students were not as happy about it. I guess I'm curious, how does this conversation about grades maybe relate to all of this? Is there any sense of grades that can be a good measurement of learning and how do we think about that in context of policies now?

Jenay Robert: Liz has so many thoughts. They're spelling out of her ears. I can actually-

Liz Norrell: You can read it on my face, can't you? Yeah.

Hans van Oostrom: Go ahead.

Liz Norrell: I started at the beginning saying that I'm not sure that it's possible to design a perfect measurement of student learning and that is why I engage in collaborative grading with my students because they have access to more information from their brain than I do. So just kind of think-- No, I don't kind of think. I really think that grades are kind of stupid. They completely pervert the incentive structures of learning in the way that Hans has been talking about where we want students to be excited about learning. We want them to be interested and to gather the skills to be lifelong learners and grades just mess all of that up.

I worry a lot about how the way we're reacting to AI is creating distrust between faculty and students in the same way that that kind of policy from Harvard creates distrust between faculty and students because faculty are saying too many students are getting A's and students are saying, "We're the best students in the country. Shouldn't we be getting A's? If you're doing your job, shouldn't we be showing evidence of learning?" These things make it harder to encourage students to do the difficult work of learning, not easier. And to the earlier point about the disconnect between executive level staff on campuses and maybe frontline staff educators, et cetera, just want to put in a plug for something that I've been doing my whole career, which is I think every one of us in a leadership position should be taking classes on our campus regularly, even if we're just auditing, even if we just pop in from time to time, pick a class outside of your discipline and go put yourself in a beginner's mindset in a class on your campus because that will help you create the empathy that I think we need for the students on our campus in a way that very little else will.

So I'm all the time taking classes way outside my discipline. I don't know how to be a good anthropologist. That's not my background, but sitting in a class about ethnography taught me a lot about how to do my job better. So I just really think that that's important.

Sophie White: Love that. I love that too.

Hans van Oostrom: Yeah, I do too. I have the same sentiment with grades, but you have to realize too that it works both ways. I think students want A's because they want that perfect GPA because they think that's how they get their jobs. This is not how you get jobs and how do we change that mindset? I've had some of my faculty in my department use mastery grading where basically there's a bunch of topics in the class related to the outcomes that you have to master and there's assessments for that and there's not a grade per se. You just either master it or you don't master it and you should master all of the topics at the end of the class. And the students hated it. They say, "I need to know if I'm carrying an A. " And so we have that whole mindset of these grades and that's what you're hearing with the Harvard students too.

They must have an A because if they don't have a perfect 4.0, they're not going to go into that next level or go to medical school or get this job. I think there's a misconception on how that actually factors in and a lot of the people that make decisions on the next level are not trusting grades because the grade inflation has already been happening for decades. And so they go, "Yeah, of course everybody has a 4.0." I just don't think that disrupting that by what Harvard is doing is helpful at all. Energy would be better spent trying to, again, come back to authentic assessment and has the student mastered the learning outcomes of the course, which I bet half the people don't know what their learning outcomes actually are. And if they do, they don't agree with them because they've been written fifty years ago and they don't like them.

Liz Norrell: Interesting though Hans, medical school is one of the places where grades have been most effectively gotten rid of entirely.

Hans van Oostrom: Absolutely.

Liz Norrell: And so if our goal is to prepare students for medical school, then maybe we should experiment with not having grades.

Hans van Oostrom: Because they're really using much more of a mastery approach because without it, I mean, when we give somebody a B, okay, they're lacking something. Are they lacking a whole topic that is important or are they just not as good at all?

Liz Norrell: One percent of every topic.

Hans van Oostrom: Yeah. And we don't know, right? The grade doesn't tell you that. So I think you really want to know that and especially in medical school like, "Oh, I didn't learn about the lung, but it's okay. I still got a B." That doesn't make sense.

Jenay Robert: So for anyone who is listening or watching the episode who's not familiar with alternative grading models, would either of you or both of you like to throw out a, here's something you can Google, here's a resource you can go look up, let's maybe help our audience because we obviously won't have time to go into the whole like how do we grade if we don't grade sort of thing. What are some resources people might be able to chase down as follow-ups?

Liz Norrell: I would definitely encourage picking up Susan Blum's edited volume Ungrading, which has lots of information about different kinds of alternative grading. David Clark and Robert Talbert's Grading for Growth is also good. My colleague, Josh Eyler has a book called How Grades Harm Students and What We Can Do About It. I may have gotten that title wrong, but I trust you guys can find it and there's also lots of podcasts and such, so many resources. My colleague, Emily Pitts Donahoe, is writing a book about collaborative grading right now that'll come out in the next year or so. So I'm really enmeshed in this, embedded in this research literature.

Jenay Robert: And fun fact, Josh, along with Hans here, we're both on the small panel that helped as advisors to the research that we're talking about today. So I had a small group of folks weighing in on the research questions and the instrument and the report. So hopefully we did the whole panel proud in reflecting their expertise as well as our own. But yeah, just another little fun fact about Josh there and Hans, you too, of course, thank you so much for helping with that work.

Hans van Oostrom: Yeah. No, I'm glad the way it came out. I think the report is great and certainly has some eye-opening results. So I think that's always good to have for reports is, okay, well, let's think about this now. Now what? Right. Yeah.

Jenay Robert: That's certainly the goal. Thanks.

Sophie White: This is great. I love how we . . . I feel like we started in a lot of different ways that I was thinking about all of this social and cultural pressure that everything needs to be measured and quantified and there are numbers behind it. I love how at the end we're talking about like, let's go to the foundations of what does learning mean and how does it affect our society in a positive way? So thank you all for the really thoughtful conversation today and for all the work that you do to support our work in higher education. Really appreciate it.