3 Ways to Plan for Unpredictable AI Demand

Gerry Bayne: Welcome to EDUCAUSE Exchange, where we focus on a single topic from the higher ed IT community and hear insights, perspectives, best practices, and more. AI is creating a different kind of planning challenge for colleges and universities. Costs can move in opposite directions, models and capabilities can change in months. Demand is difficult to predict, and the technology institutions invest in today may not be the technology they depend on a year from now. So how do you make responsible AI investments when conditions themselves keep changing? Here are some ideas from the EDUCAUSE community. The first theme is to build for constant change.

Jennifer Sparrow:When you think about when we did the internet or if we were doing a wireless upgrade, you have a very clear sense of what those capital costs are going to be, and you can plan that out several years in advance and it can be a part of that capital plan for your institution. These are dynamic costs. We don't know what we don't know, and we anticipate that there's going to be more and more demand for these things. So we're looking at ways to, how do we help to reduce some of those costs and figure out ways to do things internally that maybe can help us still meet the demand and need, but look at how we do that in a more fiscally sensitive way.

Beth Rugg: When I think in terms of AI, long-term is anything that's more than six months, which honestly doesn't align with any of our budget cycles. I think you have to be proactive, but you have to be agile knowing that we're going to come up with a plan, we're going to try to execute the plan, and we very well may need to adjust in the middle of it.

Kyle Johnson: You can't wait nine to 12 months to make a new decision. You may have to make one at three months. So you have to have some conversations with the budget office and leaders about what this could cost and what they're willing to spend. We don't know what the models look like in two to three months. Now, certainly the providers are starting to announce some slowdown in the pace at which they are releasing them, but we literally are seeing models every month or every two months. And that means not only do we not know what their capabilities are or capacity is, but we don't know what the token costs will be to use a frontier model. We have a better understanding, I think, of what some of the base models sort of cost to do things, but a new frontier model, we're always relearning that.

Ravi Pendse: In old times or cycles where maybe every six months we'll think about this, this is literally you have to think about it every day and be on top of it. Do you have a vendor agnostic strategy for your campus? And if they don't, they should seriously consider having one. What if you build everything on top of one model and what if that model has a hiccup or that's not available for whatever reasons? You can't have your entire infrastructure that you've built of teaching and learning and so on collapse because you put all your eggs in one basket.

Gerry Bayne: Our second idea is to match the model to the need.

Beth Rugg: Who really needs access? Usually the token-based consumption, it's more around the frontier models, the faster models, the higher end models. Does all of our community members need access to those tools or is it something that should be limited to particular researchers or operational needs?

Jennifer Sparrow:So one unknown is what is our token consumption going to be? What does that look like if you're a student of the arts versus an engineering student? If you're a researcher doing big data versus a researcher that's doing maybe social science research, we don't have any models to look at right now to say, this is what the cost estimates are going to look like.

Kyle Johnson: Pick the model, the smallest or cheapest model that will absolutely meet your need. So don't use the frontier model for every single thing you do, and it means we've got to educate people on what those models are good at and then teach them how to be good at asking questions because you can ask a very complex question that will cost you a fair amount just to ask, nevermind get the response. But if you can really focus the question well, then you get a sort of a cheaper back and forth.

Ravi Pendse: The way you build your harnesses, the way you provide resources to campus, they should be modular. Depending on what you're asking, it will route to the right place and candidly, it'll use the most inexpensive model if it can do the job. Also, sometimes it'll use our own internal data for which we don't pay anything. The faculty member may come in and say, "I want the best and the greatest LLM that's out there because that's what I think I need." And then when you have a conversation with them, it may turn out that that's indeed what they need. Then you of course work to provide it and it'll be expensive. You let them know about what the cost would be and if they have grant funding, then the grant funding would have to pay. There is some negotiation that occurs. Other times it turns out that what they need could be solved by something much cheaper and then you provide that to them.

Ravi Pendse: And I think often most faculty colleagues will understand the economics part of this. And as long as their research need is being met, their teaching need is being met, I think frankly, I don't think they will necessarily be vetted to one frontier lab model versus another.

Gerry Bayne: And our third theme is to align AI investments with institutional priorities.

Ravi Pendse: Sometimes what happens is you also have a situation with people say, "I need more, I need more." Getting underneath as to what is it that you're trying to accomplish, right? Having that conversation with researchers is important, with faculty colleagues is important, with administrators is important. And how does that connect with your university's core values, goals, whatever the case might be. Bringing all that conversation together requires engaged leadership from CIO, and that's really, really important.

Jennifer Sparrow:As many folks at the university community engage in how you're thinking about and planning for this, allow the faculty to speak for their needs to the provost and to your EVP and to your CFO, because this is not simply a technology ask. This is an enabling piece of the work that our faculty and students want to do. We have to be able to educate our students to be ready for the jobs that don't even exist yet. And if we don't have that faculty voice at the table, I think we lose an opportunity for this to be just an IT project and it's not just an IT project.

Kyle Johnson: There are things that we have elected to fund centrally, and that's an institutional commitment. So while it is in IT's budget line, we are not absorbing that cost the institution is investing. So there's a set of things that we will provide to folks at no cost, but then there are capacities where somebody comes and says, "I've got a grant, I have this need, I have this money because we're rolling out a way for you to be able to get API access to models and that will be a cost recovery." Again, in that unbounded world, yeah, you can't have IT absorb all of that risk by themselves.

Beth Rugg: Where's the money going to come from? It's going to come from the strategic plan. Often those strategic plans are set at the highest levels of the institution, board of trustees, chancellor, provost, cabinet level members, understanding what is most important for each of their areas. And hopefully we have to come together and agree from there with the limited resources, what can we do? Who gets it first, which will be our student success, anything tied into our enrollment management and student success initiatives, they're going to get it first and others may have to wait. But strategically, that's what we're prioritizing and why we've prioritized it.

Gerry Bayne: Our final list includes build for constant change, match the model to the need, and align AI investments with institutional priorities.