© 2008 Donald Norris, Linda Baer, Joan Leonard, Louis Pugliese, and Paul Lefrere
EDUCAUSE Review, vol. 43, no. 1 (January/February 2008): 42–67
For the past several years, EDUCAUSE publications have described the emergence of two complementary forces: (1) the growth of “academic analytics” in higher education and the knowledge services needed to support seamless sharing and leveraging of contextualized data/information; and (2) the escalating accountability demands that are driving performance measurement and improvement initiatives.1 These forces converged in the July/August 2007 issue of EDUCAUSE Review, which showcased their potentially transformative impacts on higher education.2
First, knowledge-leveraging and analytical practices are advancing in sophistication and proliferation, aided in part by a host of new software and professional services offerings. These include deploying academic analytics (tools, solutions, and services) to produce actionable intelligence, service-oriented architectures, mash-ups of information/content and services, proven models of course/curriculum reinvention, and changes in faculty practice that improve performance and reduce costs. Over time, these new offerings have the potential to support previously unattainable levels of measurement, comparison, and institutional interventions to improve access, affordability, and success for students.
Second, public demand in the United States is escalating for colleges and universities to measure, demonstrate, and improve performance and to provide access to this data. This demand is being driven by a variety of forces and interests. The most compelling is the stark fact that the international standing of the United States is slipping. In spite of the relative dominance of U.S. leading universities and their world-class reputations, the nation is losing ground in terms of the overall educational attainment of its population. The United States is also deficient in the across-the-economy-and-workforce competencies necessary for success in the global economy. Additionally, mid-tier institutions are increasingly at risk of falling behind U.S. and international competitors in their ability to track their performance and identify areas where they need curriculum and process reinvention and innovation. As a consequence, the United States faces projected declines in per capita income and economic competitiveness.
Taken together, the insights from these articles in EDUCAUSE Review paint a powerful portrait of the need for measuring performance and then moving beyond evidence/reporting to action. The action challenge is even broader than described, however. Performance includes both operational performance (administrative and support systems) and academic performance (design and execution of academic strategies to achieve learning experiences, outcomes, and real-world competencies). Pervasively improving performance requires coordinated measurement, intervention, and action across the entire education/workforce spectrum—from “cradle to career,” so to speak.3 Such performance improvement will require more effective articulation and transitions between learning enterprises and between learning and work. It will require earlier, more effective stimulation of learners so that they can acquire the skills essential for success in the global economy. Working across the entire education spectrum of learning/work requires new solutions and techniques, including the sharing of contextualized “actions that work” in improving performance. Clearly, new processes and performance indicators must be developed to measure the emerging life, learning, and work skills necessary for our changing world.
But how can we put more action into analytics? Six primary actions are needed to evolve from the current generation of academic analytics (tools, solutions, and services) to action analytics.4
- Focus on processes, solutions, and behaviors, not just on the acquisition of tools.
- Incorporate workforce factors in Pre-K-20 curricula and educational offerings.
- Utilize the new generation of open-architecture analytics to enhance access, affordability, and success for learners and to extend the ERP stack.
- Incorporate cross-institutional and inter-sectoral comparisons into solutions.
- Develop new practices/solutions that ensure the alignment of institutional goals, strategies, initiatives, interventions, outcomes, and measures in a variety of ways, including alignment from institutional to college to department to program levels.
- Develop organizational capacity and change culture to encourage evidence-based behavior and action-focused innovation to improve performance.
Examples of Performance Analytics
Most colleges and universities underutilize their data and analytical resources. Even though they may be awash in data, they may lack the specific information they need to identify the performance aspects that truly matter to them and their audiences—aspects that are not necessarily those captured in the key performance indicators of today’s institutions. Moreover, academic culture favors analysis over action. Traditionally, institutions have placed a greater degree of importance on reputation (prestige) than on improvements in academic performance. In practice, academic freedom too often translates into autonomy rather than accountability, making concerted action more complex. Yet in spite of these limitations, many institutions have found creative ways to measure and analyze performance and value.
An archipelago of performance-focused analytics solutions is surfacing across higher education. The most advanced solutions are currently found in for-profit educational enterprises and corporate learning, where performance analytics are a fundamental, guiding principle and practice. In addition, a cadre of leading-edge colleges and universities are deploying new practices:
- Fusing institutional research and assessment to create robust “institutional effectiveness” capabilities
- Generating the first wave of “academic analytics” focused on improving admission and student retention and related operational performance by implementing executive dashboards that provide leverage points for improving performance and accountability
- Utilizing the application of measurement, process improvement, and behavioral change to consciously stimulate a “culture of measurement and improvement”
- Reinventing articulation and transfer practices that are spreading across institutions
- Introducing learner-centric and co-curricular analytics
Further, cross-institutional performance comparison is poised to grow even more dramatically. Many community and technical colleges and comprehensive universities are using cross-sector analytics to better connect work and learning and then reflect that linkage in the competencies taught within the curriculum. Policymakers in a number of states are recognizing the need to better articulate performance requirements that span Pre-K-20 and link to workforce environments. They are considering and also enacting legislation to ensure greater communication and collaboration throughout the education pipelines.
Finally, an expanding menu of website resources are providing open access to information on performance and value for students, parents, decision-makers and policymakers, and the public at large. The website for the State Higher Education Executive Officers (SHEEO) contains a robust linkage engine for accessing local, state, and national data on access, affordability, and performance (http://www.sheeo.org/links/links_search.asp). The Pew Center on the States has created a “Chance for Success Index” that portrays the relative success of states in preparing their citizens for successful futures (http://www.edweek.org/ew/qc/2007/17csi.h26.html). The Lumina Foundation has assembled information in “What We Know About Access and Success in Postsecondary Education” (http://www.luminafoundation.org/research/what_we_know/). The Education Trust provides “Education Watch Online!” which provides state-by-state summaries of achievement and best practice examples of schools with high-risk students who are succeeding (http://66.43.154.40:8001/projects/edtrust/index.html). These online resources all demonstrate the public’s keen interest in the performance of the institutions and learning enterprises charged with maintaining work and education competitiveness.
Performance Analytics in For-Profit Learning Enterprises
For-profit higher education institutions have long been recognized as leaders in leveraging technology to create reusable content that can be embedded as part of core courses/curricula that ensure consistent standards of learning and demonstrable student learning outcomes. Additionally, for-profits tailor their services and practices to the accelerated learning needs of adults.5 These innovations have dramatically enhanced operational efficiencies and have emphasized measurable competencies and performance outcomes. The for-profits have also focused their operational policies and practices on “actions that work” for adult learners. These value-focused measures6 have enabled the for-profit providers to charge premium prices for their educational offerings.
In addition, for-profit institutions like the University of Phoenix and Capella University consistently make extensive use of artificial intelligence and predictive modeling in marketing, recruitment, and retention. They employ intrusive interventions with at-risk students using both predetermined analysis and dynamic tracking of learner engagement and performance. They have also streamlined and personalized articulation and transfer arrangements, reducing the cost and complexity of granting credit for previously acquired competencies. The for-profits are firmly linked to employers’ needs. They have developed their IT infrastructures to achieve these transformative goals, focusing on both “operational efficiency” and “customer intimacy.” They understand the needs of individual adult learners and particular employers and have tailored program and certification offerings to align with employment requirements. The for-profits have shaped their cultures around performance: consistent outcomes and personalized, competency-based learning are key elements of their “brand.”
Academic Analytics in Colleges and Universities
Many colleges and universities are deploying academic analytics. For example, Baylor University is using enrollment predictive modeling. The University of Alabama is predicting and improving student retention. Sinclair Community College is developing a student success plan (the Individual Learning Plan) and an early alert system to support interventions. Northern Arizona University is connecting resource utilization, risk level, and outcomes. Purdue University is using its course management system data to identify at-risk students and stimulate action. In addition, many institutions are utilizing predictive modeling services developed internally or provided by strategic enrollment management (SEM) companies as part of outsourced SEM services. At the grassroots level in many of these institutions, analytics are changing decision-making, planning, and resource allocation processes.7
Another variation on academic analytics is to make them mobile. For example, Park University utilizes Jenzabar’s PDA Executive Dashboard Application, which connects to Park’s ERP system and queries the data. The web page address that displays the first graphic can be accessed via a desktop or any mobile device that has web access and a browser. Thus even while away from campus, Park University’s on-the-go executives and representatives can use online analytical processing (OLAP) to have drill-down and dynamic graphical representation on mission-critical key performance indicators.8
On the other hand, many institutions are struggling with academic analytics. An EDUCAUSE Center for Analysis and Research (ECAR) survey conducted in late 2004 suggested that many colleges and universities harbor the illusion that they can achieve satisfactory academic analytics by simply bolting on, to their existing student information system, some rudimentary data marts/warehouses, report writing, and extract, transform, and load (ETL) capabilities.9 In reality, a new organizing, analytic, and presenting layer and changes in behavior and culture are needed to move from data to reporting to analysis to action.
Creating a Culture of Measurement, Performance, and Action
Advancing performance measurement and improvement in a college or university requires changing from a culture of reporting to a culture of high-agility, evidenced-based decision-making and action. Such cultural change calls for committed institutional leadership and attention to navigating change and to transforming behaviors at all levels. Across higher education, far-sighted executives are finding ways to emphasize performance, creating incentives to support innovation, fostering change in the traditional academic culture by modeling new patterns of behavior, and building new capabilities. Often, these efforts are initiated by members of institutional boards of trustees that have a strong performance orientation and/or by state-level performance-funding initiatives.10
A case in point is the board and the executive leaders of the Minnesota State Colleges and Universities (MnSCU). They first established targeted goals to accomplish the system-wide strategic plan and are now pushing for performance-based innovations. The MnSCU strategic plan mandates a robust innovation initiative, providing funding incentives and mechanisms for leveraging such initiatives, particularly ones focusing on student learning. The board of trustees established an ad hoc Committee on System and Institutional Assessment, which oversees performance efforts. For each campus, the committee adopted targeted measures that serve as the basis for the performance review of the campus president. MnSCU has deployed a performance dashboard to reflect these measures and supporting academic analytics, which are used at the system-wide and institutional levels. Currently, the chancellor and the senior vice chancellor are both being evaluated using performance goals. This system will be rolled out more extensively across the enterprise in the near future.
Another example of committed institutional leadership is the newly reorganized University of Toledo, which recently emerged as a single institution from the combination of the University of Toledo and the Medical University of Ohio. The new chancellor, who was the physician leading the Medical University of Ohio, has announced his intention to deploy, in higher education, the performance measurement and improvement techniques that have been reducing costs and enhancing productivity in medicine. The stated goal is to create a culture of performance measurement and improvement that can be used to transform the institution. Consulting teams are assisting the university to achieve productivity gains from the merger and to use performance measurement and improvement to drive process reinvention, collaboration, and innovation.
Coppin State University is another example. It has deployed an aggressive analytics and dashboarding capability, co-created with iStrategy Solutions, combining student, financial, fund-raising, and help desk analytics with tailored modules for assessment and outcomes measurement.11 This solution incorporates a series of key performance indicators and scorecard approaches that build the capacity to easily and dynamically modify variables and then to drill down to examine the individual students, courses, facilities, or programs that are the focus of the analyses. For the past two years, this capability has been utilized by a cadre of analytical users that today number 120 administrators, faculty, and staff, out of a total of 500 on campus. Significantly, these solutions are highly intuitive and user-friendly. During this time, users have been employing analytics to progressively change the manner in which data/information are used in decision-making processes, reshaping the nature and timing of those practices.
Coppin has used its new analytics capacity to pass NCATE (National Council for Accreditation of Teacher Education) accreditation, managing ninety-nine rubrics and milestones within its school of education for teacher certification alone, and it is poised to do the same for other professional school accreditations and for accreditation from the Middle States Commission on Higher Education. But Coppin’s academic analytics solutions are already having a truly transformative impact on the very heart of the university: encouraging daily dialogues among administrators, faculty, and staff regarding resources, practices, performance, and outcomes. Before Coppin implemented the analytics solution, good information was difficult to come by. With Coppin’s new analytics capabilities, decision-makers can immediately identify issues, determine the root causes, and take real-time corrective actions. The interim president/provost and the deans are actively using these solutions to frame decisions; new deans are being oriented in utilizing Coppin’s action analytics solution as they are brought on board. Progressively, outcomes assessment and performance are driving decisions and working their way into resource-allocation decisions. In this way, a culture of action-oriented performance is taking hold and proliferating across the campus.
Improving Transferability: Courses, Competencies, and Curricula
A renewed focus on improving articulation and transfer is another innovation yielding tangible value to students by reducing costs and time to degree. A number of state-level efforts seek to provide seamless articulation and transfer between institutions in state systems. But progress is slow, and many students still complain about having to repeat too many courses, even in states and institutions with extensive articulation agreements.
As a result, a number of more aggressive solutions have emerged. Pennsylvania public colleges and universities streamlined transfer among institutions in July 2007.12 The innovative 3+1 program offered by Indiana University (IU) offers a Bachelor of General Studies degree from IU to students who transfer 90 credits from a community college and then take 30 hours via distance learning.This offers obvious cost and accelerated graduation advantages and is likely to be imitated by other providers.13 Not to be left out, for-profit higher education institutions are continuously streamlining their transfer processes, identifying course competencies so that students receive credit for previous work, and creating individualized programs for completion.
Learner-Centric and Co-Curricular Analytics—for Life
The most powerful action analytics are learner-centric, focusing on issues related to access, affordability, and success for learners at all stages of their learning lives. Over time, these analytics will empower learners to take greater responsibility for their success, in collaboration with parents, teachers, mentors, and employers. As learner-centered analytics spread through portfolios and other media, these capabilities will also become more portable.
The emergence of learner-centric analytic tools is supporting student affairs divisions in developing innovative measurements to quantify the impact of programs focused on service learning, leadership development, and student engagement. The pedagogical foundation for these new measurement strategies is perhaps best articulated in “Learning Reconsidered,”which argues persuasively for the assessment of students' personal development and experiences outside the classroom as an integral part of the collegiate curriculum.14 In practice, the University of Baltimore and other leading institutions are utilizing the SA LINK system to document co-curricular “learning outcomes.” This system captures information on participation in leadership workshops, student government, and student organizations and uses dashboards to combine this data on students’ involvement with their academic and demographic profiles, providing new insights into the importance of both curricular and co-curricular development.
Action Analytics at Work
A key ingredient of action analytics isembedding workforce requirements in educational curricula. Recently, thirty members of the National Governors Association banded together, calling for greater integration of information, policies, and practices for Pre-K-20 and employment.15 Many states have been actively pursuing such ventures. Some of the best efforts focus on at-risk and underserved students.
Again, MnSCU is a case in point. Providing access and opportunity is a core strategic goal for the MnSCU system. At a time when Minnesota’s economic vitality and quality of life require greater educational attainment, too many students are dropping out of high school, graduating unprepared for college, or giving up on college altogether. A study entitled Mind the Gap, conducted by the Brookings Institution in 2005 for the Twin Cities, reinforced this point. It concluded that despite the region’s overall strength and economic competitiveness, deeply troubling and dramatic race-based and class-based educational disparities will seriously threaten the quality of life in the state.16
In response to this report, the MnSCU system developed Achievement Centers for College Enrollment and Student Success (ACCESS). Intervention strategies are based on powerful analytic measures known to make a difference in college readiness and success for underserved students: preparation, awareness, financial issues, and institutional responsibility for student success. Still, ACCESS will require better data, better analysis, and better tools to develop early intervention strategies for each student based on personalized e-folios. The centers will require resources to intervene and remediate and to sustain success. These resources can be deployed with more efficient and effective programs and services when aligned with the action analytics model.
In addition, MnSCU is looking to leverage its analytical resources and relationships to develop a loosely coupled, action analytics utility that will shape interventions based on a repository of contextualized “actions that work” to improve the performance of underserved learners across Pre-K-20 and employment settings. These capabilities will be utilized to develop the organizational capacity of all its public/private partners. This venture will leverage MnSCU’s statewide portfolio initiative (winner of an “Innovation Award” in 2005 from the Council of State Governments), which is reaching into K-12, postsecondary education, employers, and the community at large.17
Clearly, one size does not fit all in action analytics. Most institutions begin predictive/dynamic modeling by focusing on admissions and retention. Some institutions, like MnSCU, are focusing on serving the underserved. Some major research universities are most interested in improving the performance and accountability of their grants-management operations and/or in applying dynamic modeling to human resources and financial management. Other institutions begin action analytics in student affairs and co-curricular development. In many ways, action analytics is like a smorgasbord of options, all aligned with institutional goals and strategies. Analytics can be launched in specific, targeted areas and can then be expanded along new migration paths as administrators, faculty, staff, and students learn to incorporate analytics as a key element of decision-making.
Open Architecture Enables Action Analytics
To reach their full potential, the new generation of performance measurement and improvement solutions depends on the widespread dissemination, development, and adoption of open-architecture applications in higher education. The 2006 EDUCAUSE Review article “Making Knowledge Services Work in Higher Education” described the importance of web services, which use small components that can easily exchange contextualized data with one another and be mixed and matched. The University of Wisconsin–Madison, which estimates that roughly half of its IT budget is spent on integration issues and efforts, is among the many leading institutions moving to a service-oriented architecture (SOA) approach. Internationally, an initiative called the e-Framework for Education and Research is facilitating technical interoperability and SOA—also called “enterprise service architecture” (ESA)—within and across education and research. Moreover, the open source movement, based on open-architecture applications developed by collaborative communities, continues to grow and attract new adherents who are disillusioned with the cost, inflexibility, and unpredictability of the existing products available from software vendors serving higher education.18
Open-architecture approaches are opening up “the stack” of existing, tightly integrated administrative ERP applications (student, financial, human resources, financial aid, alumni, advancement) and academic ERP applications (course management and ancillary systems). New versions of ERP products, such as Oracle Fusion, are featuring a greater degree of openness.19 But tomorrow’s action analytics solutions will not be mere extensions of the ERP stack. Rather, they will leverage the ERP-based data, information, and context to drive new performance improvements, and they will reside in a “cloud” of open-architecture solutions, as portrayed in Figure 1. This cloud also incorporates potential linkages to the clouds that connect PC operating systems (e.g., Apple, Linux, Microsoft) to software services delivered on the Internet.
Click image for larger view.
This cloud is much more than an extension of the ERP stack. It serves as an organizing, analyzing, and presenting layer. It truly unleashes the power of action analytics by enabling not just power users but also individual users to access and analyze data, information, and context and to access and portray the results:
- Draw data, information, and context from a wide variety of sources. These open-architecture solutions will scrape data, information, and context from administrative and academic ERP systems and from structured and unstructured data, information, and context contained in assessment solutions. They will also feature information on institutional alignments: mission, vision, values, goals, strategies, and actions at the institutional, college, departmental, and program levels, for example. Finally, they will incorporate data, information, and context from a broad array of external information sources.
- Analyze and align data, information, and context. These inventive solutions will incorporate fresh combinations of the familiar elements of academic analytics—data marts/warehouses, ETL, OLAP, and business intelligence tools—but in user-friendly forms that are more affordable for the higher education market. Data mining and predictive modeling will be integral elements of these cloud-based solutions. In addition, these solutions will incorporate assessment management and alignment tools that enable decision-makers to demonstrate institutional alignment and context in the analytics.20 This is a critical element for seamlessly linking strategic planning, resource allocation, accreditation, program review, and continuous improvement processes and for embedding contextualized measurement in those processes. Alignment tools are especially critical to solutions dealing with (1) multi-campus institutions or systems, (2) peer institution comparisons and state-level metrics, (3) the cascading of institutional goals and strategies down to the campus, college, department, and program levels, and (4) the aggregating of measurements up to institutional totals and state levels. Moreover, the new generation of customer relationship management (CRM) capabilities will be more open and affordable and better suited to higher education. Finally, these solutions will include niche applications, such as dashboarding and alert systems tied to institutional metrics that trigger intrusive advising and intervention actions.
- Access and portray results through a combination of presentation tools: dashboard, portal, and portfolio capabilities. The presentation modes for analytics will proliferate and will provide a palette of options that can be combined. Users will be able to access high-level presentations, dynamically modify the variables of analysis, and drill down to greater detail - even to individual information/context elements.
Although this new generation of user-friendly, highly configurable, and more cost-effective open-architecture solutions is a key ingredient in action analytics, these solutions are not sufficient by themselves.21 To truly put the action into analytics, higher education institutions need committed leaders who will consciously build organizational capacity to measure and improve performance and to change organizational culture and behavior.
Measuring and Improving Performance: Changing Institutional Capacity, Culture, and Behavior
To instill a culture of measuring and improving performance, institutional leaders need to
- develop performance measurement and improvement strategies that nurture an action analytics culture and behavior;
- examine and reinvent existing business practices and processes that incorporate analytics;
- execute these performance strategies in an expeditionary manner, adapting to changing conditions and opportunities; and
- navigate and lead a change process to build organizational capacity, change the organizational culture, and foster new behaviors that both enable and reflect evidence-based decision-making and action.
The last area—changing institutional capacity, culture, and behavior—encompasses four capabilities: technology; information; analytics; and innovation. Most colleges and universities have achieved the greatest sophistication in changing technology capability, with lesser levels of sophistication and achievement in information, analytics, and innovation capabilities. For each capability, Figure 2 portrays the elements that are needed to nurture a culture favorable to action analytics. At any time, institutions are likely to be developing all four of these capabilities simultaneously, following intertwined paths.
Click image for larger view.
Technology Capacity, Culture, and Behavior
To foster an action analytics culture, institutions require a vision and strategy for the technology ecology of the future and for migration paths to get there. They then must execute that strategy in an expeditionary manner, adjusting their vision and strategy to fit changing conditions. In an increasingly open-architecture world, institutions will likely choose different long-term strategies for evolving and/or replacing legacy and/or current ERP systems. Many will pursue the strategy of “opening up” existing administrative and academic ERP capabilities and leveraging them with new, open-architecture applications and tools. Optimal action analytics solutions will depend on the right mix of “technology ecology”—a more open, integrated stack and loosely coupled cloud applications.
Technology vision and strategy must clearly convey a fundamental principle: in order for institutions to justify their investments in IT capacity, they must aggressively leverage that technology to reduce costs and enhance performance, and they must be guided by action analytics that not only enable but also provoke decision-making and action. Merely improving operational efficiency will never yield sufficient value to justify the investment.22 As we have learned, the track record of “value on investment” in higher education has been disappointing. ECAR research on the Y2K-stimulated round of ERP implementations in the late 1990s demonstrated that most institutions failed to use ERP systems to significantly affect performance, and ECAR research published in 2005 showed that business process reinvention in higher education had primarily addressed simple process-reinvention targets, taking a “good enough” posture.23 Moreover, many institutions have found the information and reporting capacity of ERP and business intelligence tools to be inadequate for action analytics solutions.24 Although some institutions have demonstrated the capacity to leverage ERP technology and drive business-process reinvention that involves cross-cutting, multifunctional processes,25 more aggressive technology visions, strategies, and execution are now required across higher education. The new generation of action analytics will lead to robust solutions that support more-pervasive process reinvention and innovation.
Information Capacity, Culture, and Behavior
Many institutions have developed a strategy for eventually providing unified data access, strong identity management, single sign-on, and reporting/information tools. Presentation and access are achieved through some combination of dashboards, portals, and portfolios. Many colleges and universities have developed academic analytics that include a collection of tools such as open-architecture data marts/warehouses, ETL, OLAP, reporting, and dashboards. Today, most institutions exist in a culture of information and reporting. To move beyond information reporting, they need to commit to collecting data on performance and developing the analytics and innovation capabilities needed to turn performance information and context into action.
As discussed, many of today’s institutions are awash in data, especially relating to past performance (e.g., information relating to historical demand for past courses or past performance patterns), but they are starved for actionable analyses, especially relating to current and near-future needs (e.g., evidence on what kinds of courses and programs will be in demand in two to three years and when to retire existing ones). As institutions develop their analytics capability, they will achieve new insights into and advances in their reporting capacity. These, in turn, will enable institutions to add new performance measures that better reflect changing times.
Analytics Capacity, Culture, and Behavior
A simple measure of an institution’s analytics capability is its demonstrated ability to seamlessly move from data to analysis to action. Most colleges and universities have developed some level of sophisticated analytics capacities to support their business operations. Leading-edge examples of performance analytics typically began with academic analytics focusing on recruiting and retention. Addressing the needs of at-risk learners with dynamic and predictive modeling and with aggressive interventions is also a good initiative to employ to introduce analytics. Performance-based executive dashboards can provide a leverage point as well. But most institutions have a long way to go toward achieving enterprise-wide alignment, greater transparency, sophisticated analyses that trigger actions, and accountability.
To move the institution from reporting to evidenced-based decision-making and action requires a vision and a strategy for using analytics to create the sort of loosely coupled, action analytics capability depicted in Figure 1. This strategy must include carefully chosen incentives to promote performance measurement, to change current practices, and to pursue innovations that will improve performance and value. Institutions that have developed successful performance analytics and the culture to support them have focused on processes and solutions rather than on technology and analytic tools. They have committed to “just do it” through particular process-enhancement and problem-solving opportunities, contextualizing measurements and analyses that improve processes and making decisions that lead to action.
The greatest challenge of all in building an action analytics culture is changing behaviors across the institution. Achieving faculty buy-in on performance measurement and improvement and the ongoing use of analytics solutions in their instructional practice is paramount to the institution’s ability to build and sustain a culture of evidence-based action. It is thus no surprise that the leaders in performance analytics tend to be for-profit institutions, where faculty do not “govern” the academic offerings. The following actions are characteristic of institutions that are successful in performance analytics:
- Leveraging academic analytics with a focus on admission and retention, with some faculty involvement
- Utilizing performance dashboards at the board and institutional executive level
- Separating business processes and academic support operations from faculty decision-making
- Demonstrating new behaviors as a result of applying action analytics during the daily dialogues among administrators, faculty, and staff regarding outcomes, performance, and funding
Institutions are discovering a variety of ways to achieve buy-in. Some begin by focusing on obvious goals, such as increasing the success of at-risk students and improving transitions between school and work. They then expand their goals to include other elements of performance such as improving recruitment, retention, and graduation rates. Other institutions deploy user-friendly analytics in ways that challenge the current institutional mindset, broaden the organizational discussions about performance, and foster behavioral and cultural change. Over time, sage leadership and commitment can develop the technology, information, and analytics capabilities of colleges and universities sufficiently to create more evidence-based and action-oriented behavior and culture.
Innovation Capacity, Culture, and Behavior
As institutions build their capabilities in technology, information, and analytics, they will become progressively more capable of measuring performance and demonstrating value. However, to genuinely improve performance, institutional leadership will need to commit to greater levels of collaboration and innovation—both inside and outside the campus boundaries. The institution’s vision and strategy for building organizational capacity must reflect this commitment.
Innovation can begin with individual initiatives, pilot projects, and process reinventions focusing on single-owner activities. Most colleges and universities favor small-scale, individual initiatives and process reinventions, practicing innovation with a lower case “i.” To enhance performance in a manner that makes a difference, institutions can use action analytics to justify successful performance-improving innovations being contextualized and replicated across the institution. This can result in enterprise-level innovation—innovation with a capital “I”—capable of supporting an institution that wants to become nimbler than competitors at exploiting opportunities to modify curriculum that can meet customers’ needs as they arise. Committed leadership can also create opportunities for cross-cutting process reinvention and innovations that improve performance.
Finally, one of the most important innovations that institutions will undertake is growing their capacity to develop and implement a set of new key performance measures that comport with the needs of learners and employers in the ever-changing world of the twenty-first century.
New Measures and Key Performance Indicators
The challenge of creating action analytics is complicated by the fact that the performance target is moving—rapidly. Students from the United States will increasingly be competing for jobs worldwide. The skills, competencies, and habits of mind required to be employable in today’s flattening world will not be competitive differentiators in the world of five or ten years from now. The exponentially changing pace of knowledge creation, sharing, and utilization will accelerate the necessity of acquiring increasingly complex, sophisticated, and applied skills in teamwork, critical thinking, communication, and aggressive perpetual learning. Put simply, today’s measures—such as grades, transcripts, resumes, and time-to-degree – are already proving insufficient to demonstrate an individual’s performance and potential.
New means are emerging to communicate and demonstrate competencies and the capacity for high-level performance and growth. Electronic portfolios are growing in acceptance in K-12, postsecondary, and employer settings and in applications that span these sectors. Rich representations of co-curricular engagement are being incorporated into transcripts and resumes. These mechanisms also facilitate more active engagement of employers with teachers and administrators to ensure that the developmental needs for success in the rapidly changing world of work are reflected in curricula and learners’ experiences.
Future institutional key performance indicators will still focus on access, affordability, and success—but in greater depth, more dynamically, and proactively. Institutions will need to demonstrate their performance on these three measures in comparison with other institutions and learning enterprises. Moreover, these indicators will be part of sophisticated balanced scorecard and strategy map applications that illustrate the interrelationships between measures and the actions taken to improve them.26 This will yield demonstrable, sharable insights on “what works” in improving performance in different contexts. Like MnSCU, institutions will need to consistently use action analytics to trigger interventions that improve value by affecting current and future outcomes and that anticipate the needs of at-risk groups, such as those who are currently the target of “serving the underserved” initiatives.
For individual learners, representations that express the richness of the information and context of their achievements and their potential for success must be enhanced by an order of magnitude. Analytics in use at colleges and universities must be able to demonstrate students’ past achievements as well as their capacity for future success in ways that highlight developmental, reflective, and representational information and context. Institutions must also adapt to employers’ increasing use of software to identify, validate, and attract talent as fast and as cheaply as possible. Additionally, alumni relationships can and must be fostered by using sophisticated data mining to chart and communicate proven learning pathways throughout active learning lives, cradle through multiple careers.
The future world of action analytics will be highly learner-centric. Learners at all stages will have a greater array of information, choices, and value propositions available to them. Good counsel from parents, teachers, mentors, and employers will be valued. Learners will have greater opportunities to shape their learning experiences and share responsibility for their readiness and success. These potential changes will profoundly affect the role of “learner” as we know it today.
New Skills and Habits of Mind for a Flattening, Exponentially Changing World
In the flat world of Thomas Friedman,27 the new skills for the global economy begin with a foundation of curiosity, passion, mental flexibility, self-motivation, and psychological mobility. Continuously adapting habits of mind and skills will enable global citizens to play a number of roles directly or through delegation and influence:
- Collaborators and orchestrators, who are effective horizontal collaborators able to operate in, mobilize, inspire, and manage a multidimensional and multicultural workforce
- Synthesizers, who create unexpected mash-ups with breakthrough results
- Explainers, who bring disparate things together and who can turn complexity to simplicity, opening the door to unforeseen synthesizing
- Leveragers, who bring together the right people, resources, and/or ideas to maximize and move beyond the current state, making technology and people more effective
- Adapters, who bring depth of skills to a progressively widening scope of situations and experiences
- “Green” people, who balance sustainability, renewability, and economic growth
- Passionate personalizers, who serve a global context in which people require or demand a personal touch, personally delivered services, and customized products
- Localizers, who understand the emerging global infrastructure and adapt all the new tools it offers to local needs and demands
Demonstrating the capacity to fulfill these roles will require new approaches to measuring and representing students’ performance and potential—approaches that highlight the learning outcomes that truly matter. Moreover, learners will need to be more assertive in taking responsibility for building their capacity to succeed. These roles will become even more complex and demanding in the face of the continuing revolution in knowledge creation, sharing, and utilization, as noted by Diana Oblinger. Extending Friedman’s argument, Oblinger described how individuals will need to demonstrate their capabilities to perpetually incorporate and interpret new knowledge by participating effectively in highly diverse, extended, and transnational teams and knowledge networks. These teams and networks not only will include individuals from all over the world but also will incorporate artificial intelligence (AI) agents and avatars, social knowledge networks, and other means of leveraging the collective knowledge capacity of the team or network.28
Portable Portfolios of Demonstrated Habits of Mind, Skills, and Competencies
In the evolving workforce environment of the future, the current concept of K-12 and/or college transcripts seems totally inadequate for meeting the needs of learners, teachers, parents, and employers. Eventually, portable, transportable, and fungible portfolios for learners will deploy action analytics at a personal level. This could engage learners, teachers, parents, and ultimately employers in meaningful conversations about the skills, capabilities, and habits of mind needed to be successful in the global workforce and about the ways to ensure that those requirements are reflected consistently in Pre-K-12 curricula and practice.
Current portfolio initiatives in postsecondary education are largely seen as specialized, departmental solutions to support programmatic accreditation and are confined within the boundaries of the campus. But a few portfolio initiatives that are combining K-12, higher education, and workforce settings may show how to break this impasse. These initiatives demonstrate how portfolios can be deployed to improve the performance of underserved students, to facilitate transitions between school and work, to communicate both academic and co-curricular accomplishments, and to enable employers to verify competencies.
Whether some version of today’s portfolio offerings or another vehicle or medium emerges, the fact remains that in the future, Pre-K-20 will need to provide far more robust solutions to communicate the richness of learners’ accomplishments. Ultimately, these tools and solutions and the processes that support them will become more powerful, more significant, easier to use, and highly flexible. Just as the health care field is actively investigating a personal, electronic health care record for life,29 so shall a lifelong personal learning and competency record become a key element of measuring and improving learners’ performance. Still, the greatest challenge will be achieving buy-in and commitment—from administrators, faculty, learners, parents, and employers—to an effort that would change the direction of performance measurement and require active participation and effort from all concerned.
New Key Performance Indicators and Action Analytics: A Future Focus
Undoubtedly, the current measures that appear in executive dashboards will morph as action analytics solutions spread across Pre-K-20 education and the workforce. In particular, measures of learning attainment and accomplishment are likely to expand dramatically, reflecting the kind of competency and capability demonstration requirements suggested by Friedman and Oblinger.
To date, most academic analytics solutions have looked backward, taking what we know about learners’ success to predict behavior in the next semester or to intervene today with at-risk students. But the new generation of students will place a high value on the deployment of forward-looking analytics to help them secure good jobs, and they will focus on their personal near-term and medium-term opportunities. For example, the University of East London put action into analytics to find out why not enough of its graduates received job offers at interviews. The actions taken and applied systematically to the entire graduating cohort—one student at a time—resulted in a turnaround from a sector-trailing position to top-quartile performance.30 On a larger scale, initiatives like MnSCU’s action analytics utility proposal have the potential to make sense of information from across systems and shape future policies, practices, interventions, and actions.
Put simply, the action analytics of the future will better assess students’ competencies. Using individualized planning, advising, and best practices from cradle to career, these action analytics solutions will align interventions to facilitate retention and transitions and will fully maximize learners’ success.
1. The term academic analytics, first used by WebCT, was introduced by the EDUCAUSE Center for Analysis and Research (ECAR) in describing the customization of business intelligence tools and practices in higher education: see Philip J. Goldstein, “Academic Analytics: The Uses of Management Information and Technology in Higher Education,” EDUCAUSE Center for Analysis and Research (ECAR) Research Study, vol. 8 (2005), http://www.educause.edu/ECAR/AcademicAnalyticsTheUsesofMana/158588. The potential impact of knowledge services in higher education was addressed in Donald M. Norris, Paul Lefrere, and Jon Mason, “Making Knowledge Services Work in Higher Education,” EDUCAUSE Review, vol. 41, no. 5 (September/October 2006), pp. 84–101, http://www.educause.edu/er/erm06/erm0656.asp. The impact of IT-enabled innovation on performance was pioneered in William H. Graves, “Improving Institutional Performance through IT-Enabled Innovation,” EDUCAUSE Review, vol. 40, no. 6 (November/December 2005), pp. 78–99, http://www.educause.edu/er/erm05/erm0564.asp.
2. The July/August 2007 issue of EDUCAUSE Review (vol. 42, no. 4) included the following articles: Brian Lamb, “Dr. Mashup; or, Why Educators Should Learn to Stop Worrying and Love the Remix”; John P. Campbell, Peter B. DeBlois, and Diana G. Oblinger, “Academic Analytics: A New Tool for a New Era”; William H. Graves, “Voluntary Counter-Reformation: Stepping Up to the Challenge”; and Bill Olivier, “Having Your Cake and Eating It: The e-Framework’s Service-Oriented Approach to IT in Higher Education.”
3. Quality Counts 2007, “From Cradle to Career: Connecting American Education from Birth through Adulthood,” Education Week, vol. 26, issue 17 (January 4, 2007), http://www.edweek.org/ew/toc/2007/01/04/index.html.
4. The term action analytics™ has been trademarked by Strategic Initiatives and cannot be used commercially without express permission.
5. Stephen R. Ruth, “E-Learning: A Financial and Strategic Perspective,” EDUCAUSE Quarterly, vol. 29, no. 1 (2006), http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/ELearningAFinancialandStrategi/157388.
6. Anna Caraveli and Donald M. Norris, “Value Added: New Leadership Goals in an Era of Accountability,” University Business, June 2006, http://www2.universitybusiness.com/viewarticle.aspx?articleid=40.
7. Campbell, DeBlois, and Oblinger, “Academic Analytics.”
8. Jenzabar Customer Success Story, “Park University: Meeting the Needs of a Global Student Body” (Jenzabar press release, 2005), http://www.jenzabar.net/news/SuccessStories/ParkUSuccessProfiles3.pdf.
9. Goldstein, “Academic Analytics.”
10. J. Bell, Transforming Higher Education: National Imperative—State Responsibility (Washington, D.C.: National Conference of State Legislatures, Blue Ribbon Commission on Higher Education, 2006).
11. Coppin State University example from Linda L. Briggs, “Mining Data to Find Gold,” Campus Technology, July 5, 2007, http://campustechnology.com/articles/48937/.
12. AcademyOne, “Pennsylvania Public Colleges and Universities Met Legislative Mandate of House Bill 185,” press release, July 2, 2007, http://www.academyone.com/portals/0/Press%20Release%20A1%20PA%20Update.pdf.
13. Ron Bleed, “A Disruptive Innovation Arrives,” EDUCAUSE Review, vol. 42, no. 1 (January/February 2007), pp. 72–73, http://www.educause.edu/er/erm07/erm0718.asp.
14. Richard P. Keeling, ed., “Learning Reconsidered: A Campus-Wide Focus on the Student Experience” (National Association of Student Personnel Administrators and American College Personnel Association, January 2004), http://www.naspa.org/membership/leader_ex_pdf/lr_long.pdf.
15. Andrew Mytelka, “Governors to Colleges: Focus on the Economy,” Chronicle of Higher Education, August 3, 2007.
16. Rebecca Sohmer, Mind the Gap: Reducing Disparities to Improve Regional Competitiveness in the Twin Cities (Washington, D.C.: Brookings Institution, 2005), http://www.brookings.edu/~/media/Files/rc/reports/2005/10cities_sohmer/20051027_mindthegap.pdf.
17. Council of State Governments “Innovation Awards,” 2005, http://www.avenetefolio.com/index.asp? Type=B_PR&SEC={6D124B20-C790-4A0A-8FBC-0A7927A450BC}&DE={A7F9D1B5-7567-4A28-AC12-3317BC3E26EB}.
18. Norris, Lefrere, and Mason, “Making Knowledge Services Work in Higher Education”; Jim Phelps and Brian Busby, “Service Oriented Architecture: What Is It, and How Do We Get One?” EDUCAUSE Quarterly, vol. 30, no. 3 (2007), p. 57, http://www.educause.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/ServiceOrientedArchitectureWha/161835; Olivier, “Having Your Cake and Eating It”; Brad Wheeler, “Open Source 2010: Reflections on 2007,” EDUCAUSE Review, vol. 42, no. 1 (January/February 2007), http://www.educause.edu/er/erm07/erm0712.asp; David W. Dodd, “Business Intelligence: A Vital New Resource for Higher Education,” College Planning & Management, August 2007.
19. Phelps and Busby, “Service Oriented Architecture,” p. 58.
20. Assessment management/context tools have been used (1) to support accreditation processes, (2) to align strategic planning, resource allocation, accreditation, program review, and continuous improvement processes, and (3) to embed measurement and corrective actions in these processes. Examples include TracDat by Nuventive, WEAVEonline, and some institutional solutions.
21. An online-only article published in this issue of EDUCAUSE Review—“Framing Action Analytics and Putting Them to Work,” EDUCAUSE Review, vol. 43, no. 1 (January/February 2008), online version (http://www.educause.edu/er/erm08/erm0814.asp)—outlines a model for understanding the evolving set of functional capabilities that an institution needs to be able to deliver related to enterprise performance improvement in the four functional areas that compose action analytics: strategic planning analytics; administrative analytics; academic/assessment analytics; and learning/career analytics.
22. Articles on value on investment include the following: Donald M. Norris, “Assuring Value from Your Technology Investment,” SCT White Paper, 2003; Donald M. Norris, “Value on Investment in Higher Education,” EDUCAUSE Center for Analysis and Research (ECAR) Research Bulletin, vol. 2003, issue 18 (September 2, 2003), http://www.educause.edu/ir/library/pdf/ERB0318.pdf.
23. Robert B. Kvavik and Richard N. Katz, “The Promise and Performance of Enterprise Systems for Higher Education,” EDUCAUSE Center for Analysis and Research (ECAR) Research Study, vol. 4 (2002), http://www.educause.edu/ir/library/pdf/ERS0204/rs/ers0204w.pdf; Robert B. Kvavik and Philip J. Goldstein, “Good Enough! IT Investment and Business Process Performance in Higher Education,” EDUCAUSE Center for Analysis and Research (ECAR) Research Study, vol. 4 (2005), http://www.educause.edu/ir/library/pdf/ers0504/rs/ers0504w.pdf.
24. Dodd, “Business Intelligence.”
25. David E. Hollowell, “Technology, New Facilities, and Organizational Realignment: The University of Delaware,” in Mobilizing for Transformation: How Campuses Are Preparing for the Knowledge Age, ed. Donald M. Norris and James L. Morrison (San Francisco: Jossey-Bass, 1997); Norris, “Assuring Value from Your Technology Investment.”
26. Robert S. Kaplan and David P. Norton, Strategy Maps: Converting Intangible Assets into Tangible Outcomes (Boston: Harvard Business School Press, 2004).
27. Thomas L. Friedman. The World Is Flat: A Brief History of the Twenty-First Century, 1st updated and expanded ed. (New York: Farrar, Straus, and Giroux, 2006), pp. 281–87. See also Thomas H. Davenport and Jeanne G. Harris, Competing on Analytics: The New Science of Winning (Boston: Harvard Business School Press, 2007), and David S. Evans and Richard Schmalensee, Catalyst Code: The Strategies behind the World’s Most Dynamic Companies (Boston: Harvard Business School Press, 2007).
28. Diana G. Oblinger, “Education in Exponential Times,” presentation, CACUBO Annual Meeting, Chicago, October 9, 2007.
29. Michael Fitzgerald, “The Business Case for Paperless Medicine,” CIO, August 12, 2006, http://www.cio.com.au/index.php/id;950362363; Thelma Leaffer, with Larry Mickelberg, “The Digital Health-Care Revolution: Empowering Health Consumers,” Futurist, vol. 40, no. 3 (May-June 2006); Steve Lohr, “Road Map to a Digital System of Health Records,” New York Times, January 19, 2005.
30. Jessica Shepherd, “Suits and the City,” Guardian, August 28, 2007, http://education.guardian.co.uk/print/0,,330635917-48826,00.html.