Something new is on the horizon, and depending on your role on campus, it might be storm clouds or a cleansing shower. Either way, no matter how hard you try to avoid it, sooner rather than later you will have to deal with metrics.
Metrics don’t have to cause fear and resistance. Metrics can, and should, be a powerful tool for improvement. Before we can intelligently discuss the proper use of metrics, however, we must come to a common understanding of what a metric is and how to create one.
What Is a Metric?
Most metrics are created, collected, and reported to satisfy a leader’s request. The leader’s role is to supply clarity and direction by providing the proper questions. Middle management’s role is to answer the questions. Metrics offer a means of providing the answers so that all involved can have faith in them.
Unfortunately, leaders often don’t know exactly what they want. Chances are you have played the Guessing Game with a leader, where the data you provided wasn’t what he needed, so he asked for different data, figuring he would know the right data when he saw it. Despite repeated failures, you continued to chase data as if all the effort invested in collecting the wrong data would eventually prove worth your perseverance. There is a better way.
Speaking a Common Language
Defining a metric requires a common language. Data, measures, information, and metrics are distinctly different for our purposes. Using an IT help desk as an example, we can demonstrate those differences:
- Data: The simplest/lowest unit available. Data represent “raw numbers” and are of little to no use alone.
- Number of trouble calls
- Number of employees
- Measures: A little deeper view that builds on the data. Measures are rarely useful alone.
- Number of calls per hour
- Number of cases closed by worker
- Information: Usually a comparison. This level of abstraction serves as a useful indicator.
- Number of calls for each hour compared to number of workers on a shift
- Average length of time to close a case, grouped by type
- Metric: Tells a complete story. It incorporates information (built of measures and data) to answer a question fully. Normally, a metric is conveyed through a graphical representation and explanatory prose.
To understand the relationships among the components that make up a metric, imagine a metric as an oak tree: it has a massive trunk, and the leaves and branches provide shade and comfort. Data are analogous to the leaves on the tree. Numerous and abundant, they are interesting to look at, easy to get, and serve a purpose; but by themselves they are not very useful and will not survive once removed from the branches.
The smallest and thinnest branches represent measures—they provide an essential connection between leaf and tree (between data and the metric), although not substantial or robust enough to create anything on their own. The thicker, inner branches, the limbs, are like information. Useful at times in themselves (for supporting tire swings or tree houses, for example), they die if taken away from the tree. Information without a connection to the trunk will fail to reach its potential.
The trunk of the tree, where you can determine its age and strength, represents the metric—a picture telling a complete story. The metric may consist of many pieces of information, derived from many measures and data. However, even the largest tree will wither and die without roots.
The roots of the tree represent the questions the metric is designed to answer. As with the oak, the roots define the type of tree it will become, where it will live, how strong it will be, and if it will survive a harsh environment. The roots are born of the original seed (need) and spread out, providing a life-giving foundation for the tree. Even if you cut down the tree, the roots will continue to spawn new growth. Unless the root question is no longer necessary (the tree is uprooted), you will continue to need data, measures, information, and metrics to feed the root need.
To aid in the process of properly building our metric tree from the roots up, rather than picking leaves (data) and branches (measures) off the ground trying to create a tree, we use an implementation guide. This straightforward template allows us to focus our energies on the root need.
Creating a Metric
To best use the concept of storytelling, metrics require a level of structure and rigor. The most effective and useful metrics are designed with the end in mind. Focusing your efforts up front ensures that you don’t waste time, money, or good will in collecting inappropriate data or in creating a flawed metric.
The implementation guide helps in the planning, documentation, and implementation of a metric. The guide takes basic components of a metric and lays them out for completion. It also holds the keys to building a successful metric. Continuing the mighty oak analogy, think of the implementation guide as a root stimulator. It consists of the following parts:
Executive-Level Summary: One to two paragraphs about the metric. Although it comes first in the guide you are putting together, we recommend capturing the executive summary content after you have completed the rest of the guide. It should include a definition, summary, and history of the metric (the question and the answer).
Purpose: The most important part of any metric. What question are you trying to answer? What is the root question? Why do you want to tell the story? What do you hope to achieve? This is so important that it is a go/no-go proposition. If the purpose is not clear, stop. If you cannot clearly define the root question and identify how you will use the metric, stop. An extra test is to identify how the metric will not be used. It helps if the person asking the question is open to the possibility that something other than a metric might satisfy the need more effectively or efficiently.
Success Key 1: If you don’t know the purpose of the metric, don’t collect data.
Success Key 2: If you don’t know why you’re collecting data or reporting a metric, stop.
Customer: Several possible, beginning with the “root” customer who provided the root question—the obvious recipient of the final metric. There are other customers for any metric, however. If you have people collecting data, it improves quality and accuracy when they also get to see the fruits of their labor. Don’t forget yourself—if your boss asked the question and you’re in charge of developing the metric, you should be interested in the answers, too.
Graphical Representation: A hard point to grasp for many. Rather than describe the data wanted, ask the customer to describe how they would like to view the story. Incorrectly focusing on the data instead of a metric focuses on the answer wanted instead of the question. Obviously, a focus on the answer biases the building of the metric and what it should explain. This leads to a tainted and limited view, which leads to chasing data. The graphic represents a guess at how to tell the story and at the charts or graphs to use. The graphic can be a trend, Pareto, benchmarking, bar, line, dashboard, or other type of representation.
Success Key 3: Don’t chase data: determine the question regardless of the answer.
Explanation: A prose version of the story the metric tells, explaining how to read the picture. Remember, this is only a guess. If you identified the root question well, the explanation of the answer should be evident.
Schedule: Large steps in the metric’s lifecycle. Will you start collecting data at the beginning of the school year, calendar year, or fiscal year? When will you make reports available? Finally, when will you stop collecting data? Or, asked another way, when will this metric cease to be useful?
Did we surprise you with that one? A metric has a purpose. Its original purpose can change or be overcome by events. A metric is not eternal, although your organization probably collects (and maybe reports) measures no longer used by anyone. The purpose has passed, but alas, the effort continues. Write an expected lifespan in this section. Explain how you’ll know when you can stop collecting and reporting this metric.
Measures: Time, finally, for the leaves and branches. Identify the specific data to be collected and used to develop the metric. Target the lowest-level view of the data. Nothing is eternal—the purpose, question, and data used to create the metric can, and most times should, change.
Collection: Time at last for the processes. Now you can document the processes and procedures used to actually collect the data. Be as detailed as possible—you are creating a guide for the collector to follow. Include the collector (person/role), source of the data, frequency of collecting or reporting of the data, and method of collection. Document the process for collecting the data.
It will help immeasurably if you can collect the data with as little human intervention as possible. Any time you can automate the collection process, do so. Not only do you reduce the risk of human error (inherent in anything humans do), you also minimize bias—intentional and unintentional. The less intervention, the less pain for your busy workforce. The less intervention, the less chance of human error in the collection.
Analysis: Now for the story. All the assumptions, constraints, and known flaws around the information go into the story. Document any formulas or mathematical equations needed. You might want to enlist a statistician. Many times the data will tell you how to proceed. If you’ve done the job well and identified the root questions, built a picture, and then worked on the data required, you can now allow the data to dictate—to a degree—what to do next.
Threshold and Target: The range of acceptability or expectation. Any results better than the threshold and below the target are acceptable. Any results below the threshold dictate further investigation to find out if the causes can be avoided or the processes improved. Any results above the target dictate further investigation to find out if the causes can be replicated or leveraged.
Lessons Learned: Time to get your money’s worth. Document your lessons learned, and plan to visit this section of the implementation guide periodically so that you don’t end up with a metric that outlives its usefulness, draining valuable resources past the need.
Proper Use of Metrics
Metrics offer a powerful tool for improvement. They can provide a vehicle for communication, insight for planning, and visibility for decision making. A well-thought-out metric can be a valuable asset for attaining goals and predicting the future. Metrics can help determine an organization’s health and whether its products and services align with the organizational mission and vision. In short, metrics can help leaders ensure they are doing the right things, the right way (preferably the first time).
With the power metrics provide comes responsibility. Let’s examine the risks. Metrics (and data to a greater degree) are extremely easy to misuse. Management could try to solve problems prematurely, making decisions without investigation. Someone can look at a metric and decide that it is the “whole truth.” Too often metrics are used to justify a personal agenda instead of answering a question. Misuse will create a culture of distrust, encourage workers to make the data inaccurate, and foster an atmosphere of secrecy and passive-aggressive behavior.
Success Key 4: It is not enough to ensure you don’t misuse data; you must convince everyone involved that you won’t misuse it.
The misuse of data can make all of your well-intentioned efforts worthless. Worse, it can make future communications, trust, and metrics difficult, if not impossible. Each step of the way, ensure that you are a good steward of the information you gather.
Metrics are only a tool, a means to an end, not the goal. Because metrics, like anything derived from information, contain a certain amount of error (variance), the only proper response to a metric is to investigate. Metrics in their fullest glory are still just indicators to help answer a question. You must be careful not to accept the metric without critical review. Before you act, before you make a decision based on a metric, investigate and ensure that it is telling you what you think it is. A metric provides additional information, in a structured format, but it is still just information—not a truth. You must investigate to find the truth and ensure that you make the right decisions.
Of course, some metrics won’t require further investigation, like a metric used to give you an additional level of comfort. Before making any important decisions, however, you should make sure that you are not only answering the question but answering it accurately and truthfully.
Success Key 5: The only valid response to data (or metrics) is to investigate!
Conclusion
We had three goals in writing this article. We wanted to:
- Introduce the concept of metrics as a form of storytelling.
- Promote the use of tools, specifically an implementation guide to provide structure and rigor.
- Raise awareness of the benefits and pitfalls of metrics.
When discussing metrics, remember the mighty oak analogy; it’s an easy and powerful way to explain the differences and relationships that exist among the components that make up a metric. When done properly, a metric tells a complete story and answers one or more questions. Since the question is at the root of the metric, start there and never lose sight of it. (See the sidebar on OIT Metrics.) The goal isn’t to develop the perfect data set or metric; the goal is to answer the question.
Metrics are never an end in and of themselves—they offer a way to focus your investigation. While nothing can guarantee success, the five success keys combined with a fully prepared implementation guide will help you avoid the most dangerous pitfalls.