Start Smart with Data: 10 Steps for Success

Within higher education IT departments are well-established partners in the use of business intelligence and analytics to provide enterprise-level data, to support institutional research, and, increasingly, to measure student success. Maturity of data governance, analytical methods, and practices in these areas has increased exponentially over the past decade or more. In contrast, incorporation of data into the decision-making of the IT department itself, as well as the operations of other administrative units, has typically been less extensive. Currently, there is a growing desire for data-informed decision making to permeate all of aspects of higher education.

You may be asking yourself, how can my unit or service area get started with data? What should I do first? The following steps provide a useful framework for introducing initial metrics into a unit, service, or area of work that does not yet have a robust data strategy in place. In this context, consider a "metric" to be any measurable indicator of health, efficacy, or progress toward a goal. These ten steps are the result of hard knocks, victories, and life lessons gleaned from well over a decade evaluating technology interventions in teaching and learning, measuring IT operations and support, and utilizing data to improve other administrative areas of higher education.

Start smart and you will end up with more than a pretty chart—you will be well on your way to integrating data into the strategic core of your work.

Start with Strategy

The first steps in defining metrics involve people more than technology or data. Start with conversations about the strategy for a unit, service, or area of work, as well as the day-to-day work to implement the strategy successfully.

1. Understand the strategy
   What is the short- and long-term strategy for the unit, service, or area of work? What are the primary business goals (not technology goals)?
2. Define what success looks like
   How will you know if the strategy is successful? What would achievement of your goals look like? What small steps indicate movement in the right direction? What would failure to achieve your goals look like?
3. Identify your audience
   Who will receive and review data? What actions are they empowered to take based on results? Does this audience understand and agree with the strategy and success criteria identified?

Reality check: If you cannot identify a clear strategy, success criteria, or an audience empowered (and willing) to act on data they receive, you need to spend more time on strategic planning before you move forward with introducing metrics. If some aspects of the unit, service, or area of work have a clear strategy but others do not, start building metrics around the defined aspects while working to bring more clarity to less-defined aspects. At the same time, you do not need to wait to have a spiral-bound strategic plan, a perfect Gantt chart, or a compelling infographic to get started with data. You can start with a good conversation, a rough sketch on a whiteboard, or any other form a shared vision takes. If you do not have a clear direction for which to aim, there is nothing to measure.

Align Vision with Data

The next steps concern identifying how well what you want to measure (based on your strategy and success criteria) aligns with what you can measure (based on your current data infrastructure and processes).

4. Define what you want to measure
   What are the best possible ways of measuring the success of your strategy? What additional information would be useful to have? What other ways could you tell the same (or a similar) story?
5. Assess the current state of your data
   What sources of data are available (from tools, databases, benchmarks, surveys, etc.)? What are you tracking now? How much confidence do you have in the integrity of the data? What is challenging/impossible to measure? What could you measure with a little additional effort? Where are data stored? Who has access? What is the process for making changes?
6. Locate areas of alignment
   What overlap exists between what you want to measure and what you can measure? Can you make changes to scope (narrowing from entire unit to one area of work, for example) or adjust what you track to increase alignment?

Reality check: If there is very little you can measure that aligns with your overall strategy, you will be better off establishing the infrastructure and processes to collect the data you want than to start with metrics that do not match your goals. Introducing metrics that do not tell a meaningful story can inadvertently motivate people to ignore data or, worse, to miss important details that would compel change—especially since people tend to gravitate toward numbers that make the status quo look good. At the same time, a partial story is a valid starting point, as long as it moves people in the desired direction.

Another reality check: If you question the integrity of your data (and you probably should), assess the causes and extent of the biggest issues. Are they so limiting the data would be virtually meaningless, or would they likely improve with more exposure? If the latter, do not wait for perfect data to begin to measure. Starting to share metrics can motivate people to correct errors and help boost data integrity efforts (especially if limitations or gaps are due to lack of compliance with existing processes or incorrect categorization). As you move forward, you can mitigate limitations in your data by providing clear definitions and letting people know that initial numbers are preliminary and subject to change.

Measure, Refine, Repeat

The next steps are about defining and testing initial metrics that are aligned with your strategy and available within your data infrastructure. This is where you start doing, rather than exploring and questioning.

7. Start small
   Select a few meaningful metrics that align with your strategy and start to report on them. Make sure you have clear definitions that allow for a shared understanding of what you are (and are not) measuring.
8. Communicate
   Discuss the larger strategy and success criteria and explain how your starting metrics show progress toward your goals. Also, be transparent about what you are not able to track and about limitations in your data. Gather feedback and seek out champions.
9. Iterate (and automate)
   Establish a baseline by tracking your metrics over a meaningful time period (a fiscal or academic quarter or longer, through an important milestone for your service, etc.). Act on constructive feedback about data integrity and suggestions for process improvement. As soon as you feel confident in your measurement process, automate it!

Reality check: Shared knowledge is key. As you are initially introducing metrics, numbers will shift, especially as familiarity with your data increases and data integrity improves. Expect some pushback as people start to see metrics, especially if they do not align with what they expect. Often, a strategic metric relates to a known problem or pain point, so it will not always paint a pretty picture from the start. Be a good partner—act on constructive feedback, but weather the grumbling. You are trying to find the numbers that make a difference.

Another reality check: If you do not automate, you will quickly lose the ability to innovate, since all your time will be spent manually pulling together data and reports.  

Keep Advancing

So, you have identified and introduced meaningful metrics, won over some champions, and are ready to rest on your laurels. Not so fast…

10. Continue
    You now have gathered momentum, and it is very important to keep it going. Check back with your strategy and make sure your metrics remain important. Start to fill in the gaps you identified along the way.

Reality check: This last step is, unfortunately, easy to forget, but it is where the most important work begins. If you have followed these steps, you have a strong starting point, but it is likely quite far from your desired state. Circle back and work to address the strategic, infrastructure, and data integrity issues you identified earlier. When you introduce meaningful metrics, data builds an appetite for more data. This is a key time to capitalize on the gains you have made and keep moving forward.

Cara Giacomini is Director of Advancement Analytics at the University of Washington.