At some point in their educations, students must learn copious amounts of information. To do this, they use a variety of well-known strategies such as study groups, note-taking services, and videotapes of lectures. In fall 2004, a group of first-year dental students at the University of Michigan (U-M) School of Dentistry asked to have all dental school lectures videotaped and recordings made available on a Web site. The students’ doubted their ability to accurately summarize in their notes the quantity of information presented in lectures. The students thought that reviewing a video recording of each lecture would help them better retain the biomedical information presented.
The Office of Dental Informatics is responsible for the development, implementation, and evaluation of learning-technology activities in the school. This includes faculty development, course Web pages, research with and about learning technology, and the formative and summative evaluation of learning technology projects. The office is also responsible for the acquisition, installation, and operation of classroom and computer lab equipment.
In response to the students’ request, the Dental Informatics group applied formative evaluation strategies to determine the ultimate solution. The group determined that podcasting (see the sidebar) audio recordings of lectures provided a better technology solution for the students’ needs than the originally requested video recordings.
Formative Evaluation and
Instructional Design
Instructional design and formative evaluation strategies are commonly used in developing instructional products, especially for computer-based instruction and other learning technologies. Formative evaluation provides information to help monitor and improve product development to ensure that it meets its intended goals.1
Instructional design consists of several stages (see Figure 1). In each stage, designers use formative evaluation techniques to obtain feedback concerning the product from clients, subject matter experts, colleagues, and learners.2 Feedback gathered via formative evaluation in one stage of the instructional design process is used in subsequent stages to help improve the product. Formative evaluation information is collected in four ways: self-reporting, observation, tests, and records.3 Self-reporting, the most commonly used method, refers to users’ directly relaying their experiences with or opinions of the product, generally by means of a questionnaire, survey, or interview with an evaluator. Observing users’ behavior and interaction with the products is another popular method for collecting information. When trying to determine whether cognitive or behavioral skills have been affected, tests are generally used. Occasionally records and documents, such as server access log files, are used to gather information on the frequency of downloads—data that can give an idea of how much a product is being used.
Click image for larger view.
In the Define stage of instructional design, developers begin defining the scope of the learning activity, identifying learner characteristics, establishing constraints, and collecting resources. During this stage, the learning technology team members, who have limited knowledge of the subject matter and the intended audience, commonly make many of the design suggestions. They gather information about the target audience’s prior knowledge, interests, and experiences with the subject matter through interviews and focus groups. This information guides design decisions.
The Design stage consists of creating a prototype, flowcharts, and storyboards. Feedback is gathered by assessing the user audience’s attention, comprehension, information retention, personal involvement, and user-computer comfort.4 These data guide revisions during production of the product, potentially eliminating costly and time-consuming changes at a later stage if the product does not meet user needs.
During the Development stage, the product is created, tested, and revised until the client is satisfied.5 Formative evaluations occasionally are used outside of the design, development, and implementation stages for a product. Often it is difficult to find funding, resources, and support for the development of a product without results showing that it will effectively accomplish its intended goal.6 The results of a formative evaluation can serve as an indicator of a product’s success with its intended audience, which in turn can tip the scale toward securing resources needed for further development.
Instructors often hesitate to integrate new products or technology into their courses without evidence that it will benefit student learning. Information retrieved from a formative evaluation can help them determine whether a product should be implemented.
Once the product has been deployed in its intended setting, formative evaluation can serve several different purposes. Developers can use feedback to make small improvements that were not anticipated during the initial development, for example. Feedback can also help steer future iterations of the product. If the product will be used in a different environment or with a different audience, information gathered during this part of the evaluation can guide reconfiguration of the product.
Formative Evaluation: A Case Study
Formative evaluation strategies are used routinely when developing computer-based instructional programs such as patient simulations or tutorials. When students requested that lectures be videotaped and made available on a Web site, they didn’t get an immediate decision to grant or deny the request. A research university such as the U-M values inquiry. This culture made it natural to apply strategies to systematically determine whether videotaping was the best solution. We conducted a series of three pilot studies, using formative evaluation and instructional design techniques to guide the process. Flagg7 described four types of formative evaluation measures, two of which we used in this project—self-reporting and records. No tests were administered, nor were observations of students conducted.
Students initiated and supported the project, so were equal partners from the beginning in the formative evaluation process. An Advisory Group consisting of five dental students and one representative from Dental Informatics was formed to direct the project. The staff member supported the project’s technical activities.
The Advisory Group decided to conduct a pilot in single course, with the goal of determining whether video recordings would be the most beneficial media format. The group chose six questions to answer before making a final recommendation:
From the beginning we expected that one pilot could not answer every question. At the same time, we knew we needed answers to all the questions to reach a complete solution.
The results of the first pilot would determine the project’s direction and yield additional questions. The first pilot focused on answering question 1, the second pilot answered question 2, and the third and final pilot answered question 3. We examined questions 4, 5, and 6 across all three pilots.
Certain constraints placed on the project contained costs and ensured student involvement. First, students were responsible for obtaining instructor permission to record lectures. Second, students had to provide their own playback devices. The school provided technical support. A university grant from the provost’s office funded the few additional expenses.
Pilot 1—Media Format
The focus of this pilot was to answer the question, What is the best media format for lecture review?
Pilot 1 Methods. The Advisory Group selected part of a microbiology course for the pilot because of the difficulty of the content and the dependence on diagrams and other visuals during the presentation. Faculty permission was obtained to record the lectures and post the electronic presentation files on the course Web site.
The microbiology course met three times weekly for a total of 3.5 hours per week. The Advisory Group discussed possible media formats and chose three types: (1) video, (2) audio synced with the images from a PowerPoint presentation, and (3) audio only. The Dental Informatics staff member attended and recorded each lecture, using a digital video (DV) camera.
Each resulting DV file was exported as a video file, audio synced with PowerPoint slide images, and saved as audio only. Two days after the lecture, these files were posted on the course Web site, created using CTools. The U-M’s course management software, CTools was developed using the open source content management system Sakai. Because of the university-imposed file size limit, only links to the files were posted in CTools; the School of Dentistry stored the media files on its QuickTime streaming server. It took the technical lead on the project approximately 3.5 hours to record, complete the postproduction process, and post the files for each hour of lecture.
The self-reporting measures for learning needs were a 12-question survey administered to the entire class (N = 105) one week after the pilot concluded and a focus group of six students immediately following the survey. For records, we looked at server logs to gauge the frequency with which students used each of the three media types. Finally, records were kept of the time spent by the technical staff on the project in order to calculate cost.
Pilot 1 Results. This pilot focused on determining the best media format for lectures (question 1). The three formative evaluation measures—student survey, focus group, and server logs—showed that students preferred the audio-only format. Of a possible 105 participants, 30 downloaded media directly from the Web server. Server logs revealed that 20 percent of downloads were video, 14 percent audio synced with PowerPoint, and 66 percent audio only. The average time from posting date to download was 16.2 hours. The nature of electronic files makes it impossible to determine usage among those who obtained the media in other ways (from a friend burning a CD, for example, direct file transfer from another user, or media used in groups).
Of the 105 students in the class, 70 (66.67 percent) completed the survey. Due to the formative nature of this project, we report only the percentage of students responding to a question. In Table 1 the second column reports the percentage of survey respondents who selected that answer to the question, and the third column shows responses from students who reported using the lecture download system.
Click image for larger view.
The results from the students who used some form of media are clear and confirmed in the server log records. Responding users both preferred (66.1 percent) and used (66.1 percent) audio-only over both the requested format of video and PowerPoint synced with audio. They primarily reviewed lectures, although a small percentage (9.1 percent) used the online lecture as a replacement for attending class. Most of the students reported using the files to study at home, but some also used them to take advantage of down time when working out at the gym (8.8 percent) or during their commute (8.8 percent) to school. Of the students who used the media, some downloaded their media of choice on a regular basis as soon as it was available (25.9 percent). More students tended to download the files close to the relevant exam (44.4 percent).
In addition to an analysis of the server logs and the survey, the focus group provided more in-depth comments. Four of the focus group participants downloaded and used media files; two did not. The focus group discussion probed in more detail the questions asked in the survey. Comments from the group corroborated findings from other sources, specifically that students preferred audio as the media choice.
The focus group also helped us further explore some of the survey findings. For example, in the survey an overwhelming number of users (87.3 percent) reported the CTools Web site easy to use. When we explored the same question in the focus group, however, they revealed that downloading is a cumbersome method of acquiring files from numerous course Web sites. They suggested a number of possible improvements (an issue addressed in the third pilot). Focus group participants also reported that the most popular playback devices were personal computers and iPods.
Results of the survey, consultation with the Advisory Group, and responses from the students in the focus group helped us address which courses, if recorded, would most benefit students (question 4). Survey options focused on information-dense course content with heavy reliance on visuals. The course selected by the most students (48.3 percent), histology, involves great detail and a large number of diagrams. This meant that they could concentrate on what was being said during the lecture about the visual materials instead of trying to be stenographers capturing all the information being presented. The audio recording served as a safety net that enabled students to listen to the lecture repeatedly for information they didn’t record in their notes during class. They identified the next course for recording, biochemistry (45 percent), based on the faculty lecturer’s fast speaking rate. The fast-paced delivery made it difficult to record all the important information in their notes, so the ability to review the lecture was extremely important.
To determine the support costs in terms of staff time and workflow (question 5), the technical staff member documented the time spent recording, processing, and posting the files. Recording time for one hour of class plus 2.5 hours for processing time (from DV to the three formats) was 3.5 hours total staff time. Additional time costs such as equipment set-up were not tracked.
Question 6 asked whether the number of students participating warrants the cost of the project. The dental class has approximately 105 students. Participation in the first pilot was solely by word of mouth. The server logs revealed that approximately 30 students (29 percent) participated voluntarily in the pilot, which we considered successful. Also, the four students in the focus group who had used the recordings were adamant about how helpful they found the recordings.
Pilot 1 Conclusions and Decision. The results of the first pilot clearly indicated students’ preference for the mobility of the audio recordings rather than video. The overwhelming support for audio combined with the low cost of producing the audio files compared to video resulted in the decision to commit to the project and to conduct subsequent pilots to answer the technical questions about acquisition and dissemination.
The Design stage of instructional design relies on feedback to eliminate expensive mistakes. Formative evaluation strategies used to gather student feedback early in this project provided critical information that redirected the focus of the content delivery. The result was a satisfactory and cost-effective solution requiring less technical support than a video solution.
At this stage we had identified a workable, though incomplete, solution. Having found students’ preferred media choice for reviewing lecture content, we could have stopped our formative evaluation at this point, adopted the chosen system, and attempted to scale its implementation school-wide. However, questions remained about how best to capture and disseminate these audio recordings.
Pilot 2—Acquisition Solutions
For clarity, we describe pilots 2 and 3 as two separate pilots although they ran in parallel. After pilot 1 revealed students’ strong preference for audio, the logistics of acquiring high-quality recordings of lectures at a reasonable price became the focus of pilot 2. This pilot included two courses, for a total of six class hours each week.
Pilot 2 Methods. From the beginning, we attempted to contain costs. Because the iPod would be a low-cost solution, we explored it first as an audio capture device. Students reported using iPods to record lectures, and a few students placed iPods with supplementary microphones on their desks in the front row of the lecture halls. This method produced unsatisfactory audio quality and was highly dependent on lecturer position. Because of this, the Office of Dental Informatics used a Belkin Universal Microphone Adapter to connect an iPod directly to the lecture hall’s amplified PA system. Students immediately reported the resulting audio was of extremely poor quality and almost useless. The inability to accurately monitor audio levels for the iPod along with the iPod’s limited recoding frequency (16-bit mono, 8 KHz equivalent to analog telephone quality) resulted in extremely poor recordings.
Next we explored using a computer to capture the audio—an Apple Powerbook G4. The analog audio signal from the classroom’s PA systems was fed into a computer and captured using Apple’s QuickTime Broadcaster. The Dental Informatics staff added metadata (date, course name, instructor, and lecture title) to the completed recording and posted the file to a Web site.
Using a computer for both media capture and processing streamlined the workflow (see Figure 2). This reduced the time associated with capturing, converting, and posting files. Additionally, eliminating video as a delivery format removed the need to transfer media from DV tape to computer.
Click image for larger view.
The most expensive component of any technology project is the technical staff’s time. Thus, pilot 2 attempted to automate the recording process. Once we finalized the computer platform as the acquisition solution, we turned to automating the work performed by the technical staff. AppleScript scripts were written to automate the recording, file processing, and uploading workflow. We modified the processing script to also create an Advanced Audio Codec (AAC) file as an audio option for download. AAC is a subset of MPEG-4 and allows for several advanced features like bookmarking and playback speed changes. It is also the native format for the iPod, which many students reported using.
Five students who volunteered to manage these tasks were trained in the three-step process. At the beginning of a class lecture they selected an icon from the computer’s desktop that ran a compiled AppleScript application. This launched QuickTime Broadcaster and began the recording using presets designed by the technical staff. The script required student action at the end of the lecture by displaying an “Add to iTunes” dialog box. Once the student clicked “Add to iTunes,” the script requested metadata (lecture title, lecturer’s name, and course) and added further system-generated metadata (date and time). The file was then transferred to a processing machine that automatically converted the recording to both audio formats and uploaded it to the Web site. After the class ended, the audio files were posted to the Web site.
The second pilot used two formative evaluation strategies—self-reporting and records. The self-reporting measures included a focus group and student e-mail notifications of problems. For records, we examined server logs and kept records of the time spent by technical staff on the project in order to calculate cost.
Pilot 2 Results. While the computer solution was more expensive than using the iPod for audio capture, responses from students and staff clearly indicated the superior audio quality using a computer to capture the audio from the room’s public address system. That answered question 2, about the best solution.
To determine costs in terms of staff time and workflow (question 5), we added up time required for the steps involved. Reducing the number of media formats lessened the recording, processing, and posting time to two hours per class hour. Automating the process of converting and posting files using Apple’s AppleScript technology also speeded the process. Staff processing time dropped to 15 minutes a week (mostly maintenance on processing machines), and files are available on the Web within five minutes of a lecture’s conclusion.
Pilot 2 Conclusion and Decision. Now that more information has been published about portable audio and iPods, high-quality audio production is clearly the most critical component.8 While the tools used to create the audio files (see the sidebar) are becoming easier to use, the process still requires professional technical expertise.
The results of the second pilot clearly indicated that a low-cost computer could easily capture high-quality audio recordings of classroom lectures from the room PA system. In addition, automating the recording, processing, and posting workflow greatly reduced staff time. These two low-cost solutions meant that the school could afford to sustain the project over time. The automated process combined with student support allowed the project to scale up to the point that all lecture halls now are equipped with lecture recording capabilities.
We could have stopped the evaluation here. While the solution did not represent the eventual final choice of podcasting, it did meet the students’ desired media format, allowed them to review lecture content, and eliminated expensive staffing costs.
Stopping at this point would leave several student needs unexplored, however. Specifically, students requested that we examine ways of automatically notifying them of updated content and provide better tools to navigate available files, along with enhanced features for working with long audio recordings.
Pilot 3—Dissemination Solutions
The final pilot focused on improving the process by which students identified and downloaded new recordings. Pilot 3 used the formative evaluation strategies of self-reporting and records: feedback from the focus group and Advisory Group, and review of the server logs.
Pilot 3 Methods. After the first pilot, the Advisory Group selected a course in which the faculty member spoke fast. The Advisory Group also requested recordings of the Integrated Medical Systems (IMS) series because of the large amount of difficult material it contained. Students felt that having another source to review material would be especially beneficial in this class. IMS lecture series met several days each week, for a total of 15 hours—a three-fold increase in hours recorded from the first pilot.
Once lectures from multiple courses were available, a focus group revealed that students would prefer to go to a single Web page to access the lecture recordings instead of moving between numerous course Web sites. Thus, access through the course management system, CTools, was discarded in favor of a custom-built dynamic Web site with data contained in a MySQL database. This allowed students to more easily sort files by name, media format, and class.
Pilot 3 Results. We answered question 3, the best way to disseminate the media, by consulting with the Advisory Group and obtaining feedback from the focus group. The students reported equal use of MP3 and AAC files, accessed both on personal computers and iPods. Examination of server logs confirmed this, showing equal downloads of MP3 and AAC files. In the focus group, students reported using the audio book feature of the iPod to speed up or slow down a lecture. Students specifically pointed to this single feature that made the iPod more useful than other audio players.
The Advisory Group concluded that the custom Web site, while an improvement over the CTools site, still was not the easiest method for obtaining the files. Feedback from the focus group revealed that students wanted a centralized Web site to more conveniently access the audio files. While the course Web site served adequately for one course’s lectures, the increasing number audio lectures made it more difficult to access new content.
The focus group also suggested creating either a notification system using e-mail or a subscription service for automatic notification when files were posted. We added RSS (Real Simple Syndication) so that students would not have to check for new files. Server logs subsequently showed that 50 percent of files were downloaded via RSS.
Server logs served to measure student participation and thus help us determine whether the number of students participating warranted the cost of the project (question 6). The logs indicated no difference (30 students) at the end of the second pilot but an increase to 60 students (of a possible 105) at the end of the third pilot. Students reported that the convenience of obtaining files via RRS increased the likelihood that they would download files. Server logs confirmed this, revealing a much higher download volume compared to earlier in the project.
Pilot 3 Conclusion and Discussion. The third pilot gave solid answers to the project’s final two questions. Because students used both MP3 and AAC formats equally, the decision was made to provide both formats. Data also indicated that if lecture recordings were convenient—from a central Web site and/or through RSS—a significant number of students would use the functionality voluntarily. As access became more convenient, student usage of the lecture recordings increased from 28 percent to 57 percent of students in the class using the service. The marked increase in users and usage helped the school commit to the project.
Implementation Realities
The three pilots answered the six initial questions and solidified the U-M School of Dentistry’s commitment to offer podcasts of lectures as a teaching and learning service. The final step was to make the service routinely available. Two very significant issues remained, however: faculty support for podcasting, and the institution’s ability to sustain podcasting into the future.
While the faculty involved in the three pilots were enthusiastic about participating in a research and development project, having lectures routinely recorded and distributed raises issues of intellectual property. Conflicts between a lecturer’s intellectual property rights and the need of students to acquire media for review must be mediated in a way acceptable to both groups.
Once the school chose podcasting as a routine service, the pilot’s makeshift authentication and authorization process needed to be replaced by a very robust system. Through collaboration with the U-M ITCS, the School of Dentistry adopted the U-M’s already implemented authentication and authorization system, cosign. This guaranteed that only U-M dental and dental hygiene students could access the lecture podcasts and eliminated the potential intellectual property issue, thus helping with faculty acceptance.
While the U-M School of Dentistry had proven that lecture podcasts could be done for a reasonable cost, the developed system was proprietary to the dental school. To guarantee that podcasting would be sustained at a reasonable cost over an extended period of time and that it would continue to evolve, it needed to be offered university-wide. Ideally, the knowledge, processes, and services developed within the School of Dentistry would be shared with the university community and beyond. In partnership with Apple Computer, Inc., a follow-up pilot investigated whether iTunes U could expand on the services provided by the School of Dentistry. While still in progress, that pilot’s initial results have proven positive, and the U-M and Apple Computer are now investigating integrating iTunes U into Sakai. If that project succeeds, then the School of Dentistry will use the Sakai podcasting software, guaranteeing that podcasting lectures will occur routinely for an extended period of time and at a very low cost.
Faculty and students are interested in exploring the expansion of podcasting services as a separate project with its own analysis, design, and development cycle. It is important to draw a distinction between this deliberate project expansion9 and scope creep. Scope creep refers to uncontrolled changes in a project’s goals that cause the project to drift away from its original purpose. Our purpose is to continue to research ways to aid student learning.
We will evaluate requests for new features with two intentions: that all improvements assist student learning, and that all future developments can be integrated into Sakai. For example, the processing scripts are being written to easily support complementary media components (PDFs, PowerPoint files, images) that can be associated with a lecture’s audio file.
Lessons Learned
The unanticipated results of this project strongly reinforced two lessons that can be applied to most learning technology projects: (1) the importance of actively involving the client, and (2) the importance of using proven instructional design and formative evaluation techniques. The delivery mode of podcasting lectures (audio only) did not match the students’ initial request. Indeed, stopping after any of the pilots would have met some, but not all, of the students’ needs as uncovered in our formative evaluation. An interim solution also would not have encompassed both the user needs (audio format, automatic download, easy browsing) and the institutional needs (automatic recording and processing, full integration into existing technologies) as podcasting did. By involving students in the design process and applying proven formative evaluation methods, the school avoided implementing a system that would have cost more and might not have met student needs as well as podcasting lectures.
The students’ commitment through the entire project was evidenced by their enthusiastic participation in focus groups, on the Advisory Group, and on the survey. From the beginning, students knew they were guiding the development of the project, and they sustained their energy and commitment through completion. They now share in the pride of seeing how their efforts benefit their classmates and the classes of dental students following them. They also share in the presentations, press releases, and writing about the project, as in this article.
Technology is commonly implemented into teaching and learning situations without using instructional design and formative evaluation strategies or involving students. Faculty and administrators (developers) usually make the decisions. Unfortunately, many novel and innovative projects do not succeed or have disappointing results. Using both the formative evaluation strategies suggested by Flagg10 and the instructional design process described by Alessi and Trollip11 helped us identify and adjust to unexpected circumstances and develop a successful technical solution to a learning dilemma. Ultimately, use of these strategies provided the critical data required to ensure long-term and ongoing support.