Given the hundreds of millions of dollars spent each year on technology initiatives in higher education, questions about the value of these expenditures almost seem moot. However, virtually every CIO at every institution of higher education is asked to report the return on investment (ROI) of technology spending on his or her campus. Making such an accounting is a daunting challenge. Applying a strict business-school definition of ROI would require providing evidence of a dollar-for-dollar return on hardware, software, and infrastructure projects. The desired return in question, however, is not always so obvious. Should technology investments save money? Increase institutional capacity? Improve throughput?
In the face of such ambiguity, many CIOs throw up their hands in frustration. Bob Weir, vice president of Information Services at Northeastern University in Boston, asserted that technology ROI is “impossible to reliably calculate, compare, or claim” in higher education.1 This is so, he argued, because there is no universally recognized “coin of the realm” for assigning value to IT investments.
While the academic community has become accustomed to the “ROI question” as it relates to IT in the broad sense, the burgeoning scope and expense of instructional IT raises the question in new, more targeted ways. Virtually every university and college in the United States has implemented a course management system (such as Blackboard, Desire2Learn, Sakai, or WebCT). At the same time, most institutions have ramped up their support for instructors interested in developing online courses or multimedia enhancements for traditional courses. As a consequence, institutions now face the dilemma of storing and managing terabytes of teaching and learning content. Together with expenditures on e-portfolio solutions, online assessment and evaluation tools, and increasingly sophisticated classroom technologies, colleges and universities are spending a growing percentage of their overall IT budgets on what might be termed the teaching and learning infrastructure.
Just as it is difficult to demonstrate ROI for broad IT initiatives, it is difficult to show ROI for money and time spent building, implementing, and supporting a teaching and learning infrastructure. In this article, we propose an alternative method for accounting for the costs and benefits of investments made in teaching and learning technology (hereafter TLT). More specifically, we recommend that institutions focus on measuring the value on investment (VOI) of their instructional technology programs and initiatives. For reasons detailed below, we believe this approach is both more realistic and more helpful in terms of prioritizing and assigning scarce resources to maximize institutional effectiveness. A focus on VOI allows institutions to begin with the end in mind—that is, to design, build, and implement a teaching and learning infrastructure that demonstrably and consistently contributes to the realization of desired institutional outcomes. We believe that using the approach recommended here not only makes it possible to demonstrate the value of teaching and learning investments but also makes the value more obvious and easier to communicate to others.
VOI versus ROI
Perhaps because most colleges and universities have departments or schools of business, higher educationhas a powerful tendency to demand evidence of dollar-for-dollar returns for time and money expended on IT. While this seems a straightforward proposition, such efforts generally falter in the face of competing definitions of “return.” Indeed, the lack of consensus about how to measure ROI has led many to give up on precise, quantitative measures of the value of IT investments. Similar frustration in the corporate world prompted Gartner to challenge organizations to focus on the VOI of so-called soft initiatives that do not obviously or directly add to the bottom line.2 While value can be had from investing in organizational competencies, new methodologies and capabilities, and better organizational collaboration, the dollar-for-dollar return on such investments is difficult to measure.
Stated in the simplest terms, VOI differs from ROI in its focus on intermediate rather than final outcomes. A VOI approach to IT, for example, emphasizes the contributions of new hardware or software to institutional competencies. Targeted competencies are those shown by experience, evaluation, and research to contribute to an institution’s performance—its ability to produce key deliverables.
A VOI-driven resource allocation and evaluation process promotes a broader, more strategic view of project consideration and prioritization. ROI might offer greater precision, but it also creates a tendency to emphasize narrowly tactical IT.3 Although tempting, this approach is a mistake because ROI is much more difficult to demonstrate for strategic investments aimed at broad organizational competencies and capacities. Because these soft improvements are critical to the success of institutions of higher education, however, we believe that it is at least as important for organizations to account for VOI as to account for ROI.
A focus on value instead of return promotes more meaningful measurement of the benefits of investments in TLT, but measuring VOI is not without pitfalls. The most significant problem stems from the fact that VOI is not a direct measure of an investment’s impact on the bottom line. For institutions of higher learning, the most important measurable results are things like the number of students served, time-to-completion, graduation rates, and success of graduates (initial job placement, career performance, and so forth).
Adopting a VOI-centered approach to TLT is particularly appropriate given the difficulties associated with consistently measuring teaching and learning outcomes. For example, one possible indicator of student success is average GPA. However, with grade inflation and inconsistent grading standards across disciplines and institutions, this is not an entirely valid or reliable measure. Using exit exam scores, job placement statistics, or other measures would be similarly problematic. Each institution and each program within each institution has its own set of success criteria for students, a reality reflected in current accreditation standards. Investments in TLT do not always contribute to these things in an obviously direct way. Rather, these investments contribute to institutional capacity to produce these results more effectively and consistently.
It might not be obvious how a CMS, for example, contributes to an institution’s time-to-completion or graduation rates. While an ROI model would require demonstrating exactly how a CMS contributes to these things, VOI simply requires that we show how a CMS contributes to important institutional competencies (Gartner’s soft infrastructure). Appropriately used, a CMS can promote better teacher-student communication, enhance large lecture courses, make classes more flexible for students, and so forth. These things can, in turn, improve time-to-completion and graduation rates.
The more realistic (though messier) reality of VOI is illustrated in Figure 1. If we think of the inputs and outputs of a university, the inputs are dollars, faculty, and enrolling students. The outputs are graduates, research results, and general community improvement. In between, however, are thousands upon thousands of decisions about what to do with inputs to get the desired outputs. Focusing on VOI is synonymous with focusing on how to use inputs more effectively to promote better (both in quantity and quality) outputs. The middle of the diagram depicts this as a process of prioritizing the use of inputs to increase competencies and capacities in areas that contribute to desired outputs. When inputs are used to increase the ability of an institution to produce desired outputs, we can say that the inputs (investments) have been used to add value to the institution.
Click image for larger view.
Pursuing VOI at Brigham Young University
The late 1990s saw considerable administrative support for infusing technology into teaching and learning at Brigham Young University (BYU). Behind this support was a desire to make teaching and learning more effective, to provide learning opportunities for more students, and to save money. There were significant points of disagreement, however, about which goals were most important and how they should be pursued. From an instructional technology perspective, difficult questions were raised about resource allocation and project prioritization. Should we focus on online course development? Building a sophisticated learning management system? Buying and implementing a less sophisticated (but more practical) CMS?
University leaders decided it was important not just to answer these questions but also to create a framework for answering them consistently in the future. Several key stakeholders and a new teaching and learning with technology “czar” (the TLT Czar) were assigned to tackle these issues. The brief retrospective of this effort provided here might give the false impression that we had a clear idea of the steps we would take from the day the journey commenced. That is simply not the case. Rather, we began by focusing on what seemed to be the most important first question—What is it that we really value?—and then moved on from there. The process is represented in Figure 2.
Click image for larger view.
Identifying and Agreeing on Teaching and Learning Values
The first step in strategically mapping an institution’s teaching and learning infrastructure is to reconcile current activities and projects with the institution’s mission and objectives. While it might seem grandiose and pedantic to make such a connection, doing so is vital in establishing a successful VOI process.
Our process began with a year-long project to define the university’s distributed learning strategy, starting with revisiting the university’s founding documents. Through extensive conversations with stakeholders from across campus and with input from an outside higher education consulting group (Collegis Eduprise), we articulated a strategy for teaching and learning improvement at BYU.
The strategy consisted of five specific goals:
These objectives have since been amended and fine-tuned. We incorporated the Sloan Consortium’s Elements of Quality in Online Education4 standards into our evaluation of a TLT initiative’s value, for example. Changes made collaboratively have contributed to the maintenance of a commonly agreed upon, unifying statement of TLT value.
Note that none of our TLT value objectives refers explicitly to university outputs (graduation rates, for example). Rather, the strategy emphasizes areas of opportunity where effective TLT integration can improve the university’s capacity to achieve its loftier goals.
Coordinating Support Organizations
Once articulated, these objectives became a common touchstone for organizations across campus that support teaching and learning. This in turn prompted a willingness to coordinate and collaborate in new and important ways. Under the direction of the TLT Czar, a loose federation of support organizations was established, called the Teaching & Learning Support Services (TLSS) Group. Group membership consisted of leaders or decision makers from the library, the IT office, independent study, copyright, and the Center for Instructional Design. At first, the group focused almost exclusively on learning about member organizations—what they did, how they did it, and why they did it. As areas of overlap or common purpose emerged, group meetings rapidly evolved into coordination and collaboration sessions.
The power of the TLSS was not simply open communication, however, but communication with a purpose. Each member group had bought in to the five objectives of the distributed learning strategy, so the communication (and subsequent collaboration) of the TLSS focused on better achieving these goals as an institution. Most importantly, the strategy provides a consistent, shared definition of value and serves as a foundation for a consistent framework for evaluating, approving, and prioritizing new TLT initiatives.
Creating an Environment for Collaboration and Coordination
Supported by a unified strategy statement and consistent goals, the TLSS organizations collaborated and coordinated in ways not possible a few years earlier. Strategic budget plans were shared and reviewed openly at group meetings; resources (budgets and personnel) were transferred to address institutional priorities; old programs were altered to foster better coordination; and new programs were created to take advantage of new organizational synergies.
One of the most significant accomplishments of the TLSS was the establishment of a unifying TLT infrastructure blueprint (see Figure 3). The schematic represents the group’s efforts to accomplish goals 4 and 5 in the distributed learning strategy: “Unify production and delivery of instructional materials across campus” and “Effectively manage teaching and learning resources.” The blueprint serves as a basis for rationalizing the teaching and learning systems the university adopts and supports. It also provides various groups a common framework for thinking about components of the infrastructure and how they affect other organizations. This significantly reduces the temptation to freelance and build or buy applications that meet local needs but tend to thwart enterprise interoperability and collaboration.
Click image for larger view.
This blueprint has the virtue of focusing on the functions necessary to extend and improve teaching and learning, not specific applications. For example, it is critical to manage curriculum at the university and program levels. Technology solutions can make this a more efficient and effective undertaking, certainly, but the key functional needs are the most important variables in the analysis, development or acquisition, and implementation process—not specific applications or tools.
Similarly, the blueprint identifies content management, content transformation and publication, course content delivery, and other functions that must be supported by the teaching and learning infrastructure. This non-application-specific approach keeps the focus on adding value to teaching and learning activities at the university and mitigates inevitable tendencies toward tunnel vision and vendor-induced myopia. The blueprint thus serves as a teaching and learning strategy reinforcement mechanism, constantly returning focus and attention to the value (defined by shared goals) of current and potential teaching and learning applications and tools.
Again, note that the common framework for TLT investments (as represented in BYU’s schematic) does not explicitly focus on the university’s key outcomes or results. The purpose of a common framework is instead to focus resources and efforts on improving the institution’s capacity to deliver these results. The VOI of a TLT expenditure is measured by its direct contribution to the institution’s statement of teaching and learning competencies.
Institutionalizing a VOI-Driven Process for TLT
These three steps (agreeing on common values, getting organizations on the same page, fostering better collaboration) were essential preconditions for implementing a VOI-driven process for identifying, prioritizing, and completing TLT projects that consistently add measurable value to the university. All of these efforts would have been for naught, however, if we had not also implemented a new process for evaluating, approving, prioritizing, implementing, and evaluating TLT projects. We recognized the necessity of creating a commonly adhered to, consistent framework for assessing the value of TLT efforts and expenditures at all stages of a product’s life-cycle.
In our experience, a disciplined, rigorous process is essential to the institutionalization of a VOI-driven approach to TLT decision making. At BYU, we instituted three specific support disciplines to create and sustain a culture of VOI: product management, priority management, and project management.
Product Management
Each application and tool in the TLT infrastructure is a product defined by function, not vendor. Each product has a natural life-cycle that must be carefully managed. Product management oversees all aspects of a product to ensure it meets immediate user needs as well as the university’s long-term goals and objectives. Product managers work closely with key stakeholders to evaluate the potential or current value of each product in the TLT portfolio. Product managers function most effectively when integrated into the organizations that implement the products they manage.
A product portfolio manager takes the larger view of several product managers’ products, the interactions of those products, and the value the portfolio adds to the university. Both product managers and product portfolio managers monitor the TLT environment, the effectiveness of the current configuration of products, and the value the products individually and collectively add to the university. Product managers initiate project proposals when existing products need to be upgraded or replaced or new products need to be acquired to fill gaps.
Product management is an essential support discipline because university executives responsible for TLT generally lack the time and often the product-specific expertise to monitor product spaces and make detailed recommendations about them. BYU leaders rely heavily on product managers to help them identify needs and solutions in the TLT infrastructure.
Priority Management
The number of good TLT ideas will always outstrip the resources available at any institution of higher education. Consequently, decisions must be made about the comparative value of competing TLT product proposals. The centerpiece of an effective VOI-driven process is a disciplined, value-focused priority management team (PMT). This team evaluates, approves, and prioritizes TLT project proposals. These decisions are based on a well-defined set of objective scoring anchors that allow the team to assign quantitative scores to projects based on the value they will add to the university. The scoring anchors are derived from the objectives of the distributed learning strategy, the TLT infrastructure schematic (Figure 3), and the Sloan standards. The PMT not only evaluates and approves work on specific TLT projects but also ranks and prioritizes projects to allocate resources to the most important projects first.
A PMT should consist of university and organizational leaders responsible for realizing institutional objectives affected by the product portfolio for which the team makes decisions. For example, the TLT PMT at BYU consists of the director and associate directors of the Center for Instructional Design and the TLT product manager from the Office of Information Technology. The associate academic vice presidents for Undergraduate Education and Faculty are also ex officio members of the PMT. These individuals provide executive input and endorsement of the TLT team’s decisions.
Project Management
Projects differ from products in that projects have clear start-and-stop dates and target specific, limited, product-related outcomes. For instance, a CMS is a product, but the effort to bring a CMS online in a specific hardware and software environment is a project.
Once TLT projects have been approved and prioritized, project management makes possible successful execution of initiatives in priority order. Project managers employ a set of principles, practices, and techniques that help project teams stay focused on priorities established by the PMT. In particular, project managers enforce decisions about project schedule, cost, and scope. At the task level, project managers carefully track the progress of each project and report regularly to the PMT. Project managers also optimize resource allocation and use and facilitate effective planning and collaboration for project teams.
Putting It All Together
In addition to the three specific support disciplines, a successful VOI-driven process also depends on executive-level sponsorship and participation with the PMT. Each product and project needs a sponsor—someone who has the authority to approve the product or project and to assign resources to it. Most importantly, the sponsor also confirms that a proposed product or project adds value by improving key university competencies and capacities.
The product or project sponsor may be a department chair, a vice president, or another institutional leader who champions the product or project in the context of all other competing demands for resources and attention. The TLT project prioritization framework provides the sponsor with a consistent, objective rationale for supporting specific initiatives. The sponsor anchors the product or project in the VOI-driven evaluation, approval, implementation, and evaluation process.
In addition to working closely with the organization supported or served by specific products, a product manager achieves success through his or her working relationship with a designated sponsor or sponsors. This is especially true for product portfolio managers. It is critical that these individuals partner effectively with the appropriate sponsor or sponsors having institutional stewardship for their product portfolios.
As product managers maintain a value-focused eye on each product in the portfolio, they continually consider how an established product can be optimized within the context of other university systems. Working closely with product sponsors, the product manager helps create and implement value-driven product strategies.
The product manager also works with product implementers, users, evaluators, vendors, and user communities to keep up with current local and market trends, emerging deficiencies, and opportunities. Based on these observations, product managers initiate proposals for product enhancements or replacements. As institutional requirements change (as defined by the sponsor), the product manager adjusts accordingly.
Product managers play a critical role in the VOI evaluation and reporting process by carefully analyzing the performance and financial aspects of each product, including end-user evaluations, profit and loss statements, and total cost of ownership. At BYU, product managers also work closely with TLT support organizations to conduct thorough evaluations of each product’s contributions to key institutional competencies (as delineated in the prioritization process). This process assesses the past, current, and projected future value of each product in the portfolio.5 Channeling this information back into the process permits measuring and refining the value-effectiveness of products.
Does It Really Work?
The process described above might seem too good to be true. After two years of implementing this process, however, we can declare that it does, in fact, work. An example is in order. Like many institutions, BYU has grappled with the ROI of our CMS. In the past, this question focused on the kinds of difficult-to-measure factors we have cited—does our CMS improve student GPA, job placement, exit-exam performance, and so forth? By changing the focus to VOI, we instead seek to measure the value of our CMS by measuring its contributions to our institutional capacity for effective teaching and learning.
Through surveys, focus groups, and interviews, faculty and students overwhelmingly report that our CMS enhances their teaching and learning activities. Using this data, we have been able to “count” the value of our CMS. In the context of our distributed learning strategy objectives, we can say that we have:
Considering the Sloan Consortium’s Elements of Quality in Online Education, we can also objectively demonstrate that the implementation of a CMS at BYU has:
We do not have space here for a detailed discussion of each of these assertions or the data collected to support them. Our focus on VOI, however, has allowed us to objectively evaluate the value added to the university by licensing and implementing a CMS. Through this exercise, we have determined that the value gained is worththe dollars and time required to get it. Among other things, our faculty and students report that the CMS improves their ability to communicate (particularly outside of regular class or office hours times) and to organize and manage their courses more effectively.6 Additionally, students are using the CMS to collaborate, share ideas, and complete group projects more consistently and effectively than in the past. Perhaps most significantly, growing numbers of faculty use the CMS to manage the didactic elements of their courses and low-stakes assessment (quizzes) so that they can spend in-class time on more intensive discussion, analysis, and synthesis. During the past four academic semesters, students at BYU have completed an average of more than 20,000 quizzes a day using the CMS. These and other kinds of activities are occurring on our campus at significantly higher rates because we have implemented a CMS.
A significant caveat is in order, however. While a VOI approach can help institutions avoid the pitfalls of vainly trying to measure ROI, value judgments must still be made. VOI does not promise a purely objective evaluation and decision-making process with regard to TLT. For example, we have debated (and likely will continue to debate) the dollar and time value of the benefits derived from our CMS. Are our CMS benefits worth $100,000? $500,000? $1,000,000? Another obvious factor is that wise administrators will seek to get as much value from their CMS (and other teaching and learning technologies) for as little money as possible. Consequently, we are constantly evaluating and considering alternatives to our current CMS that might return the same or similar value at a lower cost. In the end, though, a VOI approach has enabled us to make more rational, well-informed decisions about the real, measurable value of our CMS and the resources the university is willing to commit (now and in the future) to a product that produces that value.
Conclusion
We have outlined a replicable VOI process for making decisions about TLT in higher education. We readily admit, though, that no process or set of guidelines can yield a perfectly objective framework for evaluating the relative value of TLT projects. However, we believe BYU’s VOI process provides a much more consistent and objective framework than many TLT decision makers believed possible.
As you contemplate implementing a VOI process at your institution, we reiterate the importance of broad, open communication and patience. It will take time to get the right people involved in the conversation (at multiple levels both in the administration and the faculty). It is also critical to articulate and build consensus around a framework of goals against which products (and associated projects) can be evaluated and prioritized. We do not believe it is necessary to adopt the same structures and processes that we have at BYU. Nonetheless, you will need discipline, rigor, and consistency to define, implement, and sustain a VOI process that will enable decision makers to look at the same data through the same set of lenses and make widely agreed upon decisions about TLT.
Our experience tells us that with the right people involved and with a commitment to follow accepted objectives and processes, it is imminently possible for institutions of higher learning to consistently measure the relative value of potential and existing TLT products and projects. Doing so enforces a consistent focus on adding value to the core business of the university and mitigates against arbitrary and less-effective resource allocation.