We Heard You! Announcing Significant Upgrades to Your CDS Benchmarking Experience

For 15 years, EDUCAUSE has offered to its members the Core Data Service (CDS), a higher education benchmarking service that provides peer data comparisons in key IT areas. This service remains an important feature of EDUCAUSE membership for a large segment of higher education IT professionals and institutions.¹ CDS has many features and components: It is at once a survey, a series of survey reports, a data download, a reporting and analysis portal, and a peer-group management system. Members routinely rely on these features to inform business decisions and drive institutional strategy, among other uses.

With its importance as a service to EDUCAUSE members, along with the complexity it involves, CDS has been a source of both delight and frustration among members. Within the past several years in particular, EDUCAUSE has received consistent feedback from members that CDS remains important and serves a critical function but that members are facing common challenges in accessing and using their data. In other words, you the EDUCAUSE member value peer benchmarking and expect and appreciate that EDUCAUSE provides the service so cost effectively, but you want a better experience using it.

We want EDUCAUSE members and CDS users to know that we hear you and we understand your experiences — in particular, your challenges and frustrations — using the Core Data Service. We are taking some big steps forward in 2018 toward delivering a better user experience, and we want to take a moment to let you know what we've been working on and the changes you'll be seeing in the months and years ahead.

1. Member Experience

We partnered with a third-party user design firm in 2017 and 2018 to conduct extensive user interviews and testing of the CDS portal interface (the main CDS hub where members enter, analyze, and download their data). In addition to these broad channels of member feedback, EDUCAUSE regularly receives and responds to direct, individual member inquiries about CDS use, many of which revolve around members' inability to access certain data points or to successfully navigate certain features of the portal.

We used all these methods to identify several common "pain points" members encounter as they enter and use the current iteration of the CDS portal:

• Members need continuous orientation and just-in-time help with the portal.
• Members lack shortcuts to quickly access the same data they need from year to year.
• Survey items sometimes seem unclear, and members cannot trust the data for apples-to-apples benchmarking.
• The speed of the CDS portal is sometimes lagging, diminishing ease of access to the system.

2. EDUCAUSE's Response

In July of this year, you will have access to a new, redesigned CDS portal that will reflect many of the improvements and features you and other members have told us you would like to see. Based on our user interviews and portal testing, we designed and built a new environment for inputting and accessing CDS data. You will notice several improvements in this redesigned portal (as shown in figure 1), in direct response to your feedback and needs.

In addition to issues with the portal interface, we acknowledge that there are challenges with CDS data accuracy in performing apples-to-apples comparisons between institutions. Indeed, such challenges are not unique to CDS and are common in all benchmarking endeavors. Unlike alternative benchmarking services, however, CDS will continue to be a free service for members and will continue to offer transparency in its peer benchmarking — you know whom you're benchmarking against, and you can contact those institutions for more context around their data.

To address member concerns about data accuracy, this year we also conducted extensive, in-depth reviews and updates to the CDS survey instrument. Owing to the skilled work of Leslie Pearlman, senior psychometrician and researcher at EDUCAUSE, and through facilitated meetings, validation studies, and qualitative interviews with internal and member SMEs, the following improvements have been made in the 2018 CDS survey instrument:

• Educational Technology Services module redesigned, with maturity/deployment indices more inclusive of all learning technologies rather than focusing on e-learning.
• Information Security module updated to collect additional staffing and funding data, as well as to more closely align with preferred standards and frameworks.
• Analytics maturity index completely redesigned to focus on current processes, procedures, and data policies.
• A new Disaster Recovery maturity index added.

The work of systems improvement and maintenance never stops, of course, and we recognize the above highlights are only the beginning steps of a continuous cycle of listening, responding, and improving.

3. The Road Ahead

With these improvements in place, EDUCAUSE will be building toward longer-term CDS stability to ensure that this year's seeds flower into a peer benchmarking service you can continue to rely on in the years ahead.

Each year, the CDS data release cycle contains rigorous data and portal testing procedures, including development applications testing, development portal testing, and production portal testing, all with an eye toward ensuring that the data are showing up properly in the portal, that the portal is functioning as intended, and that you are able to access and use the data you need. In addition to these testing procedures, in 2018 EDUCAUSE will implement an off-cycle testing regimen, to include biweekly internal and production system checks after the CDS data go live to ensure that our team catches breaks and glitches before you do.

Finally, while we're excited to offer a redesigned portal experience this year (as described above), we want you to remain excited and satisfied with your portal experience in the years ahead. To this end, EDUCAUSE will implement a three-year repeating cycle, starting this year (shown in figure 2).

For the past 15 years, EDUCAUSE members have been able to use CDS to benchmark against peers and their own aspirations and to guide their strategic and business decision-making. We are dedicated to continuing to offer this valuable resource to our members and doing so with high-quality data and a modern, easily accessed portal environment. We hope you, the member, will continue to join us on this journey and help us make a meaningful difference in the higher education IT community!

Note

1. In the 2018 EDUCAUSE member satisfaction survey, 90% of CIO/VP members rated CDS/benchmarking as "important" for their professional roles. ↩

Mark McCormack is Senior Director of Analytics & Research at EDUCAUSE.