Online Assessment, Measurement and Evaluation: Emerging Practices
David D. Williams, Scott L. Howell, and Mary Hricko, Editors
Idea Group, Inc., 2005
$84.95 (hardcover), 300 pp.
ISBN 1-59140-747-8
Reviewed by Heather Stewart
Online Assessment, Measurement and Evaluation: Emerging Practices provides a series of studies that, considered holistically, make a strong case that evaluation as we know it must change—and is changing—as a result of the pervasive use of technology in higher education. The book examines how these changes are occurring and recommends a framework for developing and refining new practices of evaluation.
The guiding premise of the book is that evaluation is context-specific. Working online and having online tools in our midst inherently changes the context of the work and learning that we do; therefore, the means and measures must change as well. This position is underscored in Chapter 2 by Paula Doherty, who traces the Latin root of the word context to the idea of “weaving together” and draws an apt parallel to the woven context that makes up the World Wide Web.
If evaluation is context-specific—and the book offers a logical argument and structure that it is—and if technology changes the context, we are at the right time in history to begin to think critically, as the authors have, about what needs to change and what is already changing. Assuming, then, that evaluation is changing, Williams anticipates two important questions: “What is evaluation?” and “If I am developing or refining an assessment, what should I consider?” The other contributors follow this line of thinking and collectively address the question “How and why is evaluation today different from in the past?” Drawing from a vast body of literature, Williams suggests we can assume that evaluation is typically used to make judgments at a high level or across a broad spectrum. Assessment, then, is an activity that occurs at the level of student learning. Measurement in this context is a means for supporting both. Understanding the difference is helpful in determining what specifically we need to change in our evaluation strategies.
The book is divided into five sections and begins with a framework for understanding and clarifying online evaluation, assessment, and measurement. It then leads readers through four different assessment modalities and uses: Web surveys, student uses, tests, and combinations. Each of these four sections contains several thorough studies conducted at a variety of institution types in the United States and Canada, including universities, technology corporations, a commercial publishing house, and a professional society. After the introductory chapter, each of the following eighteen chapters of the book offers a different example of how the process and tools for understanding educational work and data are evolving through technology.
The book is an anthology of evaluation methods and modalities that are in some way influenced by technology, and, as in an anthology of literature, the introductory chapter is necessary reading. From there, the reader could pick and choose which sections and topic areas to read, but to understand the whole, it is best to read them all.
The editors provide operational definitions for basic terms that often cause debate within the assessment community and, as a result, are misunderstood in higher education at large. These definitions help the reader understand what evaluation is. Separately, readers are presented with a conceptual framework for defining an evaluation, which is an excellent guide for anyone crafting or refining an evaluation, whether for student learning, program review, or employee performance. The framework defines an evaluation that includes (1) context, (2) stakeholders, (3) “evaluand” (or, evaluation target), (4) issues and concerns, (5) values and criteria, (6) questions, (7) data collection and analysis, (8) reporting results, conclusions, and recommendations, and (9) implications.
A quick read immediately identifies tips and techniques for using technology to enhance evaluation. Online course evaluations, tips for using bulletin boards and learning management systems for post-and-vote peer reviews, and electronic portfolios are all mentioned, and programs and studies that use them successfully are discussed. Clearly evaluation is changing at that level.
After a more thoughtful review, however, it becomes apparent that there is a fundamental shift in the way that assessment is occurring, too. This shift is deeper because it suggests that assessment is no longer an ancillary part of education but is at the heart of the curriculum. Thinking of assessment from a constructivist perspective means that learners are involved in constructing knowledge socially and by engaging in metacognition and reflection. If this is the case, technologies such as online discussions, cybercoaching, and reflective electronic portfolios can support assessment in ways that traditional assessment methods simply can not. Here the context provides an opportunity to fill a gap in an evolving world.
Considering measurement, there are both practical suggestions and thoughtful examinations of testing concepts. For instance, there is a list entitled “Bakers Dozen Questions for Test Designers” and a section on the current use of obsolescent means of validity testing, which offers new ways of considering validity, given technology-based environments. The book contains a number of ready references that are useful for those conducting basic survey research. For those engaged in sophisticated testing and measurement, the book provides thought-provoking recasting of concepts.
In all, this is an excellent book for many audiences. For faculty, instructional technologists, technical trainers, and leaders in academic computing, it is essential that we understand the shift that is occurring in online assessment, evaluation, and measurement. Although the book reads much like a refereed academic journal, the concepts contained within and the Gestalt at the end make it worth the focus. This type of thinking about education, assessment, and technology is not historically within the domain of technologists. Our unfriendly interface with it has all too often been when we find ourselves trying to evaluate the impact of technology on education in order to prove its value. This book moves beyond that old game. It answers how our practices are changing and gives us a concise, nine-piece conceptual framework and some good examples to help us create and refine our evaluations and ways of understanding.