Central IT organizations under stress often cannot offer a high level of service to groups with above-average support needs. An example of such a group would be a well-funded, research-oriented computer science department.
Several factors contribute to the increased demand on IT organizations. Given the availability of relatively inexpensive-to-purchase computational resources and the popularity of leveraging commodity hardware into Linux compute clusters, IT support organizations face increasingly greater pressure to support computational research efforts. It turns out that the total cost of ownership of these Linux clusters is often underestimated, however, based on the relatively low purchase price.
In addition, budgets have not kept up with demand and aging infrastructure, putting a further strain on resources. This growing demand for central IT support without increasing the budget—and in the face of an aging infrastructure—limits the ability of a traditional, centralized IT organization to offer a sufficiently high level of computing support for academic research groups needing above-average levels of support.
This Good Ideas piece summarizes how departments at several research universities have addressed some of the challenges of providing a high level of service in specific computationally intensive research departments such as computer science.
Rice University
Rice University has a centralized IT (CIT) organization, typical of many smaller private universities. Networking, telecommunications, educational technology, intrusion detection, and so forth are all part of this organization. Desktop support is physically distributed across campus, reporting through a centralized management organization. Supporting platform-specific system integration teams work on common software and hardware issues such as defining standard operating system images, installing software, deploying remote management tools, and recommending hardware, all through the CIT. A description of this organization has been previously published.1
In general, the advantages of a CIT organization are easier implementation and deployment of standards, personnel backup, and load balancing while maintaining a high level of ongoing personalized services that result in better customer-service levels. It is also a fiscally efficient model because it minimizes redundancy. However, this sort of organization may not be able to support the needs of all academic units, particularly large and well-funded research groups. It tends to spread resources across the entire campus at the expense of some groups that bring in more funding and need more support. This model has neither the flexibility to dynamically reallocate funds that would mirror the grant activity of specific academic units nor the budgetary capacity to take on additional requests for premium level support.
At Rice, our current CIO will be stepping down (to focus on his primary full-time job as chief librarian and on research duties). In response to these issues, the deans of the schools of Natural Sciences and Engineering formed a special faculty committee to focus on IT support issues such as organizational and funding models.2 The stated mission of this faculty committee is to "define and understand the problems with Science and Engineering IT support and infrastructure and explore strategies and plans for improving it."
As the support manager for desktop and academic computing for the schools of Natural Sciences and Engineering and the IT representative to this committee, I was asked to study some peer institutions and report on comparable metrics and strategies to kick off internal discussions.
Method
I embarked on a process of meeting some of my peers at other highly ranked research universities to build a network (the human kind) for discussion and support. Based on my research, I have made a few observations focused on the unique funding models and the evolution of each school's support organizations.
I studied rankings and overall school metrics to identify a few key schools and find the closest person to my direct counterpart at these schools. The observed schools were Princeton University, the Massachusetts Institute of Technology (MIT), and Carnegie Mellon University (CMU). I then went to the schools for face-to-face discussions. This approach led to immediate bonding and candid conversations.
Comparison of Schools
Table 1 shows a comparison of specific metrics of the studied schools, covering size, cost of tuition, and research expenditures. Overhead is the portion collected from grants that goes directly to the university for shared costs such as office space, utilities, and the budget for the central IT organization. Total Engineering Research Expenditures is an estimate of the total amount of money spent on research activities.
Click image for larger view.
Autonomous Support Groups
When the IT support demands of any research organization outgrow what is available from a centralized IT organization, often a department-specific support group is created, a type of organization referred to as an autonomous IT support group. These groups typically need at least five full-time equivalent positions to maintain critical mass and offer service levels differentiated from the central organization.
Although Rice currently does not use an autonomous support group model, Princeton, MIT, and CMU have autonomous support groups. As I took a close look at these schools, with particular emphasis on how they are funded, it was apparent that schools with autonomous groups could provide higher levels of service.
Autonomous IT support groups generally share certain characteristics:
- They must offer services redundant with those from the central organization (such as remote access, backup, desktop support, and so forth).
- They may have strained relations with the central organization.
- They need significant financial and management support from the department.
- They can be early technology adopters and offer more innovative services (wireless, Blackberry, and 24 7 support) and higher service levels than offered by the central IT organization.
- The services they provide offload pressure from the central organization to perform at unachievable levels, possibly lowering overall frustration levels.
- They promote a shared environment within the supported department but not among departments. Sometimes new technologies that have been pioneered by the autonomous support group spread to other departments or the central campus.
- They have evolved over several years to their present state, rather than having been organized into the current model.
Typically, autonomous IT support groups have evolved from, and are organized around, disparate hardware and software support efforts. But they are only able to exist in an environment with significant management support. For example, faculty may not be given the opportunity to opt out of paying the fees charged by the autonomous support group for their department.
Princeton University
The Computer Science Department at Princeton has a highly innovative group—providing innovative technologies is part of Princeton's culture and charter. The employees in that group are more highly cross-trained than any other group I observed. Any system administrator can handle issues dealing with the network, database system, firewall, PCs, UNIX, and so forth. The Princeton Computer Science Department autonomous IT support group has a philosophy of resource sharing that benefits the entire department. Few services for one research group aren't also available to every other group.
Princeton has a unique situation because almost all of the CS professors live within two miles of campus. They have built their own DSL services, renting lines from the local telephone company. They offer Blackberry service—great for frequent travelers and also allows staff to stay connected while off-campus. Along with the regular services, they also offer some unique ones, including a firewall, Web cams, and security monitoring.
My favorite quote from the lab manager is, "Faculty committees are useful to provide recommendations and feedback. Committees may not be very effective in managing computing services."
Massachusetts Institute of Technology
MIT has an even more decentralized model than Princeton. Each research group at the Laboratory for Computer Science (LCS) funds its own dedicated system administrator. In addition is the Central Resource Services (CRS) group dedicated to supporting LCS, with five full-time positions.
Little formal coordination exists between the system administrators and other parts of the support organization. For example, they don't use a help ticket system and have no explicit security policy. The Lab for Computer Science at MIT has a laissez-faire policy with a lot of trial-by-fire training. Despite the decentralized approach, things seem to hold up well, and researchers appear relatively content with the service they get. Quite a few of the 20 dedicated research lab system administrators are contractors and have worked previously for the CRS group.
My favorite quote from the lab manager is, "When things go well, they wonder what they pay us for; when things break, they wonder what they pay us for!"
Carnegie Mellon University
The CS department at CMU has the largest autonomous support organization of any school in this limited study. The current director of computing facilities for CS has a unique background in that he was previously (in the 1980s) involved with the central IT organization. Having returned to CMU more than a year ago, he is in the CS autonomous IT support group this time. He supervises 42 people supporting 2,000 users and 4,500 machines.
CMU has the most elaborate accounting abilities of any studied school, which allows them to track many small transactions and attribute specific costs to platform-specific users. The accounting system is partly the result of various Defense Contract Audit Agency (Department of Defense accounting) requirements. This homegrown system is deemed necessary in raising funds to support this autonomous group properly, although it has grown to be a significant maintenance burden.
The CS department has monthly charges for user accounts to cover storage, e-mail, printing, help desk, and other services, plus additional monthly machine charges to pay for hardware contracts, software licenses, and other machine-specific costs. Given CMU's large size, faculty have the ability to opt out of some of these charges (for example, if they want to manage their own machines). Other charges, like a networking fee, cannot be avoided if the faculty member wants an account. The help desk staff actually look up users to make sure they are current on their payments before staff will address their issues.
Another unique feature of CMU's autonomous group is that it uses a centralized approach to hardware purchasing for the CS department. For example, all new Ph.D. students for the six departments in CS will get a pre-built dual-boot machine. The group builds a Red Hat image with many additional CMU-specific modifications, such as Kerberos support and linking to the university-developed Andrew File System protocol.
Funding Models
Princeton, MIT, and CMU recognized a similar need to develop autonomous IT support groups to offer high service levels for compute-intensive researchers. Nonetheless, they have developed unique funding models to pay for the extra support (as seen in Table 1). Princeton levies an equal head tax on each faculty member and each graduate student. The MIT group charges a small percentage of every dollar of grant money brought in. This way, larger grant winners, who probably need and use more resources, contribute more money. CMU has evolved a system based on user fees for everything from accounts to network connections. Machine fees are charged based on the operating system platform. But in all these methods, the source of money is research grants.
Results
Studying these peer institutions has helped the Rice faculty committee understand several alternative organizational and funding models to address increasing support needs in research-intensive departments such as computer science. We now have a variety of funding models to consider for increasing the Rice budget to allow higher support levels.
Our plan is to ramp up funding by including more computing support costs in grant budgets. These additional funds will be used to increase staff in our central organization but targeted to specific research support areas. For reasons of fiscal efficiency, we currently don't anticipate starting an autonomous support group if we can continue to work with a CIT organization.
The actual flow of funds from grants to IT budget is still under discussion. Granting agencies, for example, have requirements that must be strictly followed. The plan implementation will also be reviewed by the new Rice CIO.
The reporting organizations of these autonomous support groups vary somewhat. Some are essentially run as businesses, with the group responsible for ensuring that funds are budgeted, collected, and spent. Others have strong support from the department administration, which ensures continuity and fiscal security. Of course, any method of organization has to fit into the basic culture and fabric of the institution in order to succeed.
An additional key observation is that my peers were extraordinarily open and eager to discuss common issues with me. Without exception I was heartily welcomed, and people even went out of their way to converse with me. I encourage others in situations similar to mine to be proactive in building relationships with peers; the payoffs can be personally rewarding and immense for your organization!
Conclusions
Clearly no single solution can handle increased system administration support for research-intensive groups, whether from an organizational point of view or from the funding side, that will work universally for all schools. Nonetheless, organizations should be willing to evolve and adopt aspects of other models that best fit their needs. Each school has its own culture and unique attributes that could spawn as many unique solutions as there are groups that need them.
I learned several valuable lessons in my research:
- Often CIT organizations cannot support the specialized needs of a computation-intensive research group.
- An autonomous support group can only exist with significant financial and management support from the host department.
- A CIT organization (with no autonomous support groups) is probably forced into providing a higher level of service for the entire campus in order to support research-intensive departments, perhaps at the expense of other services.
- A variety of funding methods and allocation models are possible, but the sources for funds are all based on research grant awards.
- Successful organizations need to have the flexibility to evolve over time with particular respect to organizational and funding issues.
Rice has a predominantly central IT organization but also has some groups that demand much more support than others. We want to increase our support levels for those who need it, which requires coming up with an appropriate organizational and funding model. Other schools have successfully addressed these problems by evolving autonomous support groups, separate from the central organization. These groups are funded by fees paid out of research grants and can raise the level of service they offer, but need strong management support to ensure continued funding. Since they offer redundant services, these groups are slightly less efficient than totally central models, but that is the cost for a higher level of service.
Autonomous support groups appear to be popular in larger schools having high levels of research grant activity. Smaller schools might not have the additional funding needed to support a group large enough to act as an autonomous support group. The groups studied seem to be very successful, however, which is to say that their clients are happy with the service levels they receive.