Federal Indirect Costs Affect Total Cost of Ownership

Jerrold M. Grochow, Consultant and Senior Advisor to NET+, Internet2

Indirect cost recovery is an important total cost of ownership (TCO) factor where cloud services and on-premises services are treated quite differently.¹ Current rules require applying federal indirect cost rates to direct-charged computing costs such as cloud services but not to capital equipment, including research computers purchased as capital equipment (i.e., above a minimum threshold).² With federal indirect costs rates routinely over 50 percent for most large research universities, this is a significant factor.

When we deal with computers on campus, someone at the university bears the costs of the computers, network, space, electricity, cooling — everything that goes into providing and operating a data center. In the case of research computing, some of these costs (typically the computers and racks) are borne by individual research projects, while other costs (such as space, electricity, cooling, and computer operators) are borne by other departments and included in the university's indirect cost pool.

In the case of cloud computing, all the costs of providing the computing service are rolled into the service cost from the cloud service provider. According to federal cost accounting rules, those costs are "direct costs" to research grants and subject to indirect cost burden. This means that cloud computing service costs will increase by 50 percent or more³ as seen by the researcher (or, the researcher will get two-thirds or less of a grant to use for direct costs).⁴ Cost comparisons between on-campus services and cloud services are made even more difficult because some costs of providing on-campus systems (such as technical support) are not usually tracked to the specific research project, and some costs may be difficult to allocate, whereas the cloud computing costs are quite clear.

Furthermore, federal rules also state that even though research computers are intensive power users (certainly more than most other uses for similar floor space), they are not subject to indirect cost burden. This means that researchers do not bear the cost of space to house their computers, power to run them, or equipment (and more power) to cool them — even as part of overhead recovery (since their purchased computer equipment does not get burdened with overhead). With the significant costs to power and cool today's research computers (as much as $50,000 per year for a single rack) versus providing lighting and cooling for more typical office or research space (which can be in the range of $2 per square foot per year⁵), researchers with large banks of computers are getting a cost subsidy over their less compute-intensive brethren — as well as over researchers who are moving their work to the cloud where no such subsidy exists. How much can this impact the decision of a researcher to move to the cloud? A lot!

Calculating Costs: Researcher vs. Institutional View

Take the following example: compare a rack of servers that cost approximately $100,000 (160 cores with 128GB RAM) to an equivalent amount of computing capability from Amazon AWS. Amazon's cost calculator says that it would cost approximately $90,000 per year to provide equivalent AWS at current rates.⁶ While it may seem that the on-campus hardware is significantly less expensive than the cloud solution, the institutional TCO has to include software, hardware maintenance, power costs, additional hardware to make the computers usable (the racks themselves, network switches, PDUs, etc.), and other costs of managing racks, bringing the on-campus solution up to about $130,000 per year (based on a three-year computation). So AWS might be seen as some 30 percent less expensive.⁷ But from the point of view of the typical researcher making the decision about how to buy computation, adding in institutional overhead gives a different picture.

While hardware isn't subject to overhead recovery, other costs are — if they are considered "direct costs" to the research project. In this case, space, electricity, cooling, networking, and the general costs of operating the data center are not direct costs, so the on-campus solution would not have any overhead (or very little) added for its computation capability. On the other hand, the full amount of the cloud service is considered a direct cost, so it would incur overhead. If overhead is 50 percent, the cloud solution is now $135,000 per year and significantly more than the on-campus solution as seen from the researcher's point of view (the research only pays for the on-campus computers and not for most of the ancillary costs).

What we have here, of course, are the different points of view of the institution and the researcher. In effect, the researcher is getting a significant subsidy (from the institution) if he or she chooses the on-campus solution. Although most institutions recognize this issue, they typically leave this decision up to the researcher. University officials and corporate groups alike have raised this disparity to the Office of Management and Budget, but the possibility of changing the regulations is unknown at this stage.

Total Cost of Ownership Analysis

The ECAR-TCO Working Group has identified indirect costs as well as other hidden costs and subsidies that affect TCO (for example, some types of departmental labor such as the graduate student who helps out with computer support), centrally borne costs such as business continuity services, and many types of infrastructure costs.⁸ The purpose of a TCO analysis is to understand the total cost to the institution of providing a service, and not just to a particular researcher or department's budget. It becomes particularly important to identify costs that are not directly accounted for in the university's or IT department's accounting system (i.e., "unhide" them) to complete the analysis. While it is true that subsidies from one to group to another "don't affect my budget," they are all costs to the university; if the real TCO of a service can be highlighted, the university has the opportunity to direct resources to the one with the lowest TCO — and more money to fulfill its educational and research mission.

1. See also the September 2014 EDUCAUSE Review Online blog post, "Hidden Costs and Subsidies Complicate TCO," which discussed related issues in performing complete TCO calculations to compare the costs of on-premises versus cloud-based computing solutions.
2. For many years, this has been OMB Circular A-21, now being replaced by OMB Uniform Grant Guidance [https://www.whitehouse.gov/omb/grants_docs].
3. Typical indirect cost rates for a large research university are in the range of 50–75 percent.
4. Again, this is from the researcher's point of view. Indirect costs are real costs of operating the university, and indirect cost allocation is the way that the university recovers these costs.
5. This pamphlet and summary are among various online resources on managing energy costs in office buildings.
6. AWS comparison was done with Amazon's TCO calculator.
7. The Amazon TCO calculator makes various assumptions about different costs, and users can change these to fit their specific situations. I used the default for these calculations.
8. The ECAR-TCO Working Group constituted last year is producing a report that will detail the various costs and other considerations in determining total cost of ownership both for locally and cloud-sourced services.