It's Not the Change, It's the Difference: Evaluating Technology on Campus

min read
It’s not the Change, It’s the Difference: Evaluating Technology on Campus
To determine the effectiveness of a new technology implementation on campus, measure the difference it makes to your constituencies

"It’s not the fall, it’s the sudden stop." This common aphorism highlights the importance of results, not just the transition from one state to another. I propose a parallel axiom for a university considering or planning a change in its technical resources: "It’s not the change, it’s the difference."

Technology affects increasing segments of the professional and personal lives of all members of the university community (faculty, staff, administrators, and students). Technology links individuals to their studies, research, and work as well as to other individuals and groups within and outside the campus community. Technology mediates the individual’s relationship to the institution itself. It takes a lot of processing power and connectivity to create and maintain such an environment.

If technology is ubiquitous, then so are users and their myriad uses of technology. IT might represent only five percent of an institution’s budget, but its impact far exceeds five percent of the institutional mission. That’s why consideration of the difference technology makes on campus is critical in planning for or evaluating technology implementations.

Look Beyond Technology

To appreciate the difference technology makes requires beginning at the end. This does not mean the point when hardware or software is up and running; rather, it means examining how a new or contemplated change in infrastructure or resources transforms the way members of the campus community act and interact—with each other, with information, with new sets of tools, with new privileges and limits. Changing the available technical resources produces the following impacts:

  • Status changes as some individuals gain and some lose access to or control over information.
  • Relationships shift as new patterns and opportunities for interactions, both electronically and physically, are added or eliminated.
  • Access to information, processes, and people changes.
  • Privileges are extended or curtailed as different sets of data become available electronically.
  • Human needs for feedback and interaction with others are addressed when isolated jobs move online or are exacerbated when new processes eliminate opportunities to interact.
  • Experience and skills change as some skills lose value to the enterprise while others give their owners enhanced responsibilities and authority.
  • Knowledge requirements change as new opportunities become available, challenging some to grow and causing others to drop out or fall back.

While many of these changes enhance or limit individuals, others have a public and organizational impact as the interplay among members of the community shifts. An interesting example is librarians. The professional lives of librarians have changed significantly with the use of the Internet and search engines. Gained was access to specialized collections, data resources, and otherwise inaccessible works for faculty, students, and staff. Lost was the personal and professional satisfaction librarians achieved from working with faculty and students to identify and locate resources. At the same time, student and faculty researchers feel liberated from trekking across campus to the library and digging through the stacks (a change they consider an advantage). When they bypass the insights and guidance of a librarian, however, researchers miss seminal works available only in print or remain unaware of works that are digitized but not accessible via Google.

Difference Information

To look beyond the technology requires obtaining difference information. This means—in addition to knowing the technical capabilities and specifications of the tools—going deeper to learn about the difference the technology makes. Instead of asking, "How has (or will) the technology change?" we need to ask, "How has the technology we already have improved or impeded teaching, learning, research, working, and living?" and "How will this proposed technology make a difference in teaching, learning, research, working, and living?" If the question focuses on the presence or absence of a particular technical capability, the response tilts toward technological determinism. When the question focuses on the difference the technology can or does make, the answer should relate directly to the mission and goals of the institution. In other words, evaluate the tool as a tool, not as an end in itself.

Although data on gains and losses are readily available, they usually get lost or overlooked in the press of dealing with immediate details and constant updates. In its 2002 report "Preparing for the Revolution: Information Technology and the Future of the Research University" (see the Further Reading sidebar), the National Research Council of the Academies of Science proposed that colleges make a concerted effort to seek out and capture data that addresses these difference factors. The goal is to have a richer understanding of how the available technology actually helps or hinders the efforts of campus constituents.

The users know exactly how a tool is being used, regardless of the vendor’s claims. They know whether a tool helps them or not, and what is missing. But without a means to capture that data, you lose the richness of knowing the difference, replacing it with the readily available measures of technical capability and availability.

This loss of difference information has pervasive consequences. Without an established procedure for capturing difference information that is reflected in budgetary priorities and IT decisions, technology departments easily slip into a focus on acquiring the biggest, fastest, state-of-the-art resources. These acquisitions provide impressive data on capabilities while ignoring the differences such technology purchases cause. Clearly the information systems department has technical expertise that parallels the expertise available in the grounds and maintenance department or in the registrar’s and development offices. Unlike decisions made by those groups, however, technology decisions reverberate throughout the campus. In addition to the effect of IT decisions on the budget, college and university administrators must consider the institution-wide differences that IT decisions cause, beyond the new capabilities added by the technology.

When trustees and legislatures investigate IT expenditures, too often the response they get stresses the increased speeds and feeds enabled without detailing how those increases made a difference (positive or negative) in the quality of work done. There are no tools or strategies in place to identify and evaluate the differences fostered. On a pragmatic, day-to-day level, without knowing the differences that technology makes, there is no way to effectively manage the changes that occur—mitigating the impact when negative and building on it when positive.

I propose a new maxim for IT: It’s not the change, it’s the difference. This axiom encourages CEOs and CIOs to look beyond the technology and consider how changes in information resources ripple far beyond the technical staff and even the specific user population targeted for a change. Establishing methods for measuring and evaluating this difference information gives administrators a clearer idea of technology’s effects on campus and helps the IT department plan its expenditures more effectively.

Frank W. Connolly ([email protected]) is a partner in ConseQ Associates and an emeritus professor of Computer Science and Information Systems at American University in Washington, D.C.