Espace

Universities in the Netherlands have increasingly introduced competency-based learning in their educational programs. This means that instead of memorizing study materials, studentsengage in active learning by completing authentic (that is, professionally relevant) assignments. The development and administration of such assignments has led to an exponential increase in the workload for teachers, however, who find themselves unable to cope. This has led to a situation where students simply do not get feedback on many or even most of their assignments, unless the assignments contribute to their grades. Even then, the feedback is often minimal (usually only a grade) and in any case arrives too late for the student to revise the assignment before turning it in.

Many authors¹ have described the potential benefits for students and teachers of using peer assessment. Students obtain more feedback on their assignments than with teacher feedback alone, especially when the teacher has little time available and when multiple peers participate. While feedback from other students may not be as authoritative as that from an expert teacher, it is available in greater volume and with greater immediacy.

Students working on assignments can use peer feedback to improve their final results or performance. As reviewers, students feel responsible for their peers' learning and are motivated to give well-informed, constructive feedback. Applying assessment criteria to essays, reports, presentations, and other assignments also results in deeper learning than from just reading texts. Both as learners working on assignments and as reviewers, students learn the peer assessment skills needed by lifelong learners, not only in their continuing learning but also in the context of professional life, where teamwork and interpersonal skills are highly valued.

Teachers face possible gains in cost effectiveness, since they can manage peer assessment processes rather than giving direct feedback to large numbers of students. Peer assessment allows teachers to assess individual students less often but more effectively and to spend their limited time on activities where teacher involvement is essential.

Peer assessment can take different forms. Oral feedback, usually given during group meetings, has the drawback that no record remains. Even if audio- or videotaped, the feedback is not easily stored and retrieved. Paper documents permit written peer feedback but require a lot of time and effort to organize. Software such as virtual learning environments can help in digitally organizing peer assessment. This kind of software does not have features to support peer feedback, however, so the teacher still has a lot of organizational work such as sending messages to students and checking messages between students. Finally, another option is to use applications tailored for peer feedback, such as Moodle Workshop and Turnitin Peer Review, or to build in-house software that automates peer feedback processes.

Four universities in the Netherlands (Open University Netherlands, University of Twente, Inholland University, and Rotterdam University) have jointly developed and tested new peer-assessment software called Espace (an acronym for Electronic System for Peer Assessment and Coaching Efficiency). From 2004 to 2006, we developed the Espace software (using Java, Apache, and MySQL) and field-tested it in a project funded by the Dutch Digital University. The main goals were to provide free licensing, modifiable open source code, and a platform-independent Web application with a built-in, easy-to-use quality-control system. The Espace system's benefits include:

• Improvement of educational quality. Students get more feedback on assignments, learn how to use assessment criteria, and learn how to give and receive feedback from peers.
• Cost containment. The system enables efficient use of available teacher time, while a built-in monitoring system safeguards the quality of peer feedback.

Espace has a number of advantages versus existing software. Compared to a commercial package such as Turnitin Peer Review, for example, Espace is inexpensive and modifiable. While open source software like Moodle Workshop focuses on quantitative feedback and summative assessment (grading), Espace focuses on qualitative feedback and formative assessment to help improve student performance. In addition, Espace features an easy-to-use quality-control system for monitoring and improving the peer feedback processes. Systems comparable to Moodle Workshop and Turnitin Peer Review have been developed throughout the higher education sector, but without the Espace quality-control features.

In the context of higher education, peer assessment refers to the critical evaluations by students of other students' products or performances, and the giving and receiving of quantitative and qualitative feedback between students or peers. Peer assessment can be summative or formative. Formative assessment is used to aid learning (learning-oriented assessment), and summative assessment is used for grading purposes (certification-oriented assessment). Formative peer assessment increases student acceptance of this process because it is nonthreatening and informal as opposed to high-stakes formal assessment. Most students appreciate feedback because it lets them improve their work before it is actually graded. Peer assessment should focus on formative assessment because summative assessment changes the cooperative nature of the peer learning relationship.²

User Roles in Espace

Because Espace is a Web application, users only need a browser. We distinguished three kinds of user roles: administrators, tutors, and students. Usually, the teacher serves as tutor, although graduate assistants could also be tutors. The administrator imports student accounts (in the form of XML files) and couples these accounts to courses. The administrator is also responsible for correctly importing assignments, assessment criteria, and feedback instructions received from the teacher (tutor).

Before a particular course with peer feedback starts, the tutor defines the settings for that session. For instance, the tutor determines when the course starts, which students will get which assignments, how many versions of the assignment outcome are obligatory, whether feedback is given anonymously or not, and so on. The tutor also decides the settings of the Espace quality system, such as what percentage of students will get extra feedback from the tutor. When the session has started, the tutor sees a to-do list as his home screen in Espace, helping him monitor the quality of the learning taking place in the course. For instance, the built-in quality system will randomly select feedback to be inspected by the tutor according to the preset percentage. We will give a more detailed description of the diverse aspects of the tutor role in the paragraph on the quality system.

The student who logs into Espace also sees a to-do list, with items such as "do assignment," "give feedback," "react to feedback," "make final version of the assignment," and so forth. A student can be an assignment maker (completer), a reviewer (feedback giver), or both, depending on what the tutor has decided beforehand (possibly in negotiation with students). This makes it possible not only for students in the same class to give feedback to assignments completed by their classmates but also for students from one class to give feedback to assignments from another class, for example a senior class to a junior class. Group work is possible, as well. A group of students completes the assignment and gets feedback from another group of students.

Figure 1 shows a student screen with some items in the to-do list.³ Clicking on the item opens the related action screen. When the student performs the action, the item disappears from the list. Possible student actions are "do assignment," "give feedback," "ask for elaboration," "give elaboration," and to produce the next or final version of the assignment outcome. If a next version is possible, the feedback loop starts over again.

Click image for larger view.

Built-In Feedback Quality System

The underlying philosophy of Espace is that the software should be user-driven and adaptable. Espace supports the teacher in her teaching and therefore must be able to support many forms of peer feedback in many educational situations. Before peer assessment can start, the tutor must set the stage. Depending on the educational situation, the teacher acting as tutor selects the options in the software, possibly in consultation with students. The main software settings are:

• Choose participants. Decide who will participate in the course.
• Compose groups. If applicable, participants can be clustered into groups.
• Select assignment makers and feedback givers. Decide which participants (or groups) will complete which assignments and which participants (or groups) will give feedback on which assignments.
• Write assignments, assessment criteria, and instructions. The instructions cover doing the assignment, giving feedback, and reacting to feedback. These texts are imported by the administrator and are reusable.
• Select the number of feedback givers per assignment. The software randomly allocates feedback givers to assignment makers. This allocation can be manually altered before and during the course, for instance, to replace a dropped-out student or to avoid conflicts.
• Select the number of versions the assignment maker can or must produce.
• Activate anonymity of student interactions. Students will not see who did assignments and who gives feedback when anonymity is activated.
• Activate student mail. To reduce his workload, the tutor can deactivate the mail option so that students cannot contact him.
• Activate elaboration option. This option lets the feedback receiver ask the reviewer for clarification.
• Set the alarm level of the outcome rating. Below this minimum acceptable rating, the tutor will be notified of a low assignment rating by a reviewer.
• Set the alarm level of the outcome rating consensus. Below this minimum acceptable consensus, the tutor will be notified of a low consensus between two or more reviewers.
• Set the alarm level of the feedback rating. Below this minimum acceptable rating, the tutor will be notified of a low feedback rating by a feedback receiver.
• Set the alarm level of repeated low feedback ratings. Above this maximum acceptable level, the tutor will be notified that repeated low feedback ratings are given to the same reviewer.
• Set the percentage of assignment outcomes to receive tutor feedback. The tutor's reviews are in addition to the peer assessment.
• Set the percentage of students randomly inspected by the tutor. The tutor inspects the outcomes and feedback of a preset number of students.

Espace's built-in quality system supports the teacher in monitoring the quality of the peer assessment taking place. Depending on the tutor's software settings, she can set the levels of quality control and time expenditure, which are inversely related. This trade-off between quality and efficiency can be changed during the course. All settings have default values that make defining a new session easy and fast.

When logging in to Espace, the tutor might find the following items in her action list:

• Low ratings of assignments
• Low consensus of assignment ratings
• Low feedback ratings
• Repeated low feedback ratings
• Give tutor feedback
• Inspect students
• Read student messages
• Deadline alerts

When the tutor opens an item in the action list, all the relevant student information appears. For instance, if there is a low consensus of outcome ratings between reviewers, the assignments, criteria, instructions, outcomes, and ratings of the involved students will be presented to the tutor for evaluation. After evaluation, the tutor can select one or more options from the following list of tutor interventions:

1. Let the feedback giver repeat his feedback. If the tutor feels the feedback giver can do a better job, she can choose to put this item back on the feedback giver's to-do list. The tutor can also send mails to the assignment maker and the feedback giver explaining why she has done so and how feedback can be improved (see intervention 4).
2. Let another student give feedback. If the tutor feels the feedback giver cannot do a better job, she can put an extra item on the to-do list of another student instructing him to give feedback. The tutor can also send e-mail to the assignment maker and the original feedback giver explaining why she has done so. Students should be informed before the course starts that they can be required to give repeated or more frequent feedback.
3. Give tutor feedback. The tutor might decide it is necessary to give feedback herself, for instance when the assignment outcome is too complex or when no students are available to give feedback.
4. Send e-mail to a student. The tutor can explain why she made certain interventions and how assignment makers (but especially feedback givers) can improve their performance.
5. Let a student repeat the assignment. This intervention is used when an assignment maker has by accident or intent submitted an assignment outcome well below the expected quality.
6. Remove item from to-do-list. The tutor can remove an item from her to-do list when she feels she has dealt with it sufficiently. This option can also be chosen without taking any other action, for example when the tutor's evaluation shows no need for intervention—it is a false alarm.

Recommendations

Espace was tested through field trials in two teacher-education courses and one general writing course. Student users said they found the tool easy to use, socially and intellectually stimulating, and helpful in leading to better assignment outcomes. One student remarked, "I was skeptical at first, but now I wish we had this tool available last year." Teacher users found that the tool saved them time, although they needed some preparation time to get to know the tool, adjust the settings, and introduce Espace to students. Time savings are thus most impressive with many student users. Of course, the less sensitive the setting for the quality system features, the less time the tutor needs to spend monitoring the peer feedback processes.

A number of considerations are important before and during the implementation of peer feedback. First, the assignment has to be such that peer feedback is useful. If there is a model answer for the assignment, students can compare their outcomes to this model and do not need peer feedback. In other words, the assignment must lead to divergent answers where useful feedback can only be provided by a tutor or by peers. Also, the assignment should not be too difficult for peers to assess. Well-defined criteria are vital for peer assessment.

Some students may not welcome the responsibility of assessing their peers or value their peers' assessments. For peer feedback to be acceptable, the educational purpose should be stressed to get students intrinsically motivated. Because peer feedback adds to the students' workload, this should be taken into account when calculating the total course workload and possibly compensated for by reducing the number of other activities. To avoid peer feedback fatigue, peer feedback should be used in moderation and certainly not concentrated in one time period.

Finally, students should not have the opportunity to be free riders and skip assignments or miss feedback deadlines. We therefore recommend that students have some extrinsic motivation to perform the peer assessment processes properly, either by earning marks for it or by making peer assessment a prerequisite requirement to finish the course.

The Espace software is available at no charge to educational institutions under an open source license agreement. For more information, see <http://www.ou.nl/espace> or contact Espace project leader Maurice De Volder by e-mail at [email protected].