Significant investments in time, money, and effort go into developing and applying technology to improve teaching and learning.1 As universities pursue such projects, they must determine the impact and value of technology for student learning.
During the past decade, funds spent on technology for educational purposes have tripled throughout the United States.2 Determining a hard return on investment (ROI) for the time and money spent to improve education is difficult, however. Institutions should also measure the value on investment (VOI) that their funds and efforts yield.3 In the study of faculty and their technology projects at Brigham Young University (BYU) described here, we emphasized VOI in terms of intrinsic factors such as satisfaction with the use of technology, increased productivity, and frequency of technology use.
Background
The Center for Teaching and Learning (CTL), formerly the Center for Instructional Design at BYU, partners with faculty to help improve teaching and learning. The CTL currently supports a broad range of faculty projects to maintain and improve on-campus instruction. It has more than 35 full-time employees and approximately 115 student employees.
Each year personnel at the CTL complete more than 180 large- and small-scale technology projects, expending a considerable amount of money and thousands of hours of labor to help faculty improve teaching and learning. Measuring the projects' impact is often neglected, however. The study described here began when administrators at BYU wanted to learn how the CTL could better serve faculty and students and best use the funds allocated to teaching and learning with technology. They felt the way to do this was to examine the impact of small- versus large-scale technology projects in terms of cost, value, and satisfaction.
Methodology
This study focused on 600 faculty who worked with the CTL on projects between 2003 and 2006. These individuals were sorted by seven project types and further subdivided into three discipline-related groups based on their college affiliation. A random stratified sample was drawn, and three faculty were selected from each of the 27 subgroups identified for in-depth interviews. We conducted 63 faculty interviews and administered a follow-up survey completed by 46 faculty (73 percent). Participants answered several questions related to how they valued their projects. Our research focused primarily on the findings from the faculty survey.
Research Context
For the purpose of this study, we define large-scale projects as typically requiring more than 50 hours of work by the CTL and small-scale projects as requiring less than 50 hours of work. In addition, small-scale projects usually cost less than $20,000 (although some can cost up to $40,000). Large-scale projects range from $40,000 to $60,000, although some have cost as much as $250,000.
Large- and small-scale projects are both important to faculty, and one type is not necessarily more successful than the other. The distinction between large- and small-scale projects is made in this study because the CTL contains two production entities and divisions of labor within those entities. For example, one entity at the CTL, the Teaching and Learning Lab (TLL), is staffed by student employees who only work on small projects—those whose scope lies within the resources available at the time (such as student skills and project loads) and that are relevant to the CTL's mission. The other entity consists of the CTL administrative personnel, who primarily work on large-scale projects requiring extensive design, documentation, prototyping, production, and evaluation.
The primary criteria for accepting a project concept proposal are project scope, resources, and schedule. The proposal review process includes estimates about the number of students and courses affected and judgments about project validity and usefulness. Projects are also prioritized in terms of need and the success of a working prototype.
Large-scale projects are typically designed for frequent use in the classroom and to impact a large number of students. Small-scale projects are used less frequently and typically impact fewer students.
Large-scale projects include Committee for Instructional and Media Arts (CIMA) and Faculty Fellowship projects. The CTL works with faculty to create CIMA projects that help solve instructional problems and enhance high-enrollment undergraduate courses. The projects' design must include significant cost savings and/or improve students' understanding of the subject matter. One CIMA project is Brain Development, an interactive CD-ROM that helps students understand and follow the developmental stages of the embryonic human brain.
A Faculty Fellowship is a one-year program in which a select group of faculty analyze their teaching and technology use and then create individual projects to improve student learning. The Psychometric Statistics Web site—one example of a Faculty Fellowship project—contains interactive lessons that give students a greater understanding of how to use graphics, text, sound bites, and bivariate statistics in psychological testing.
Small-scale projects include mini projects, personal technology training, and TLL sessions. Mini projects are created by student employees who work up to 50 hours on each project. An example of a mini project, the EV-1 Transmission Animation, shows students how a vehicle transmission works through multiple views of a transmission in motion.
Personal technology training sessions provide faculty with the opportunity to scope, design, and begin to develop an instructional media project. In half a day, faculty receive hands-on, personalized attention as they learn how to incorporate an instructional element of technology into their classes.
The TLL provides support for faculty who come to the CTL for help with CD-R archiving/burning, digital imaging, digital video/audio, and interactive multimedia (Flash, QuickTime, Blackboard, and PowerPoint).
The CTL also works with academic departments at BYU to develop university courses for independent study. Finally, technology innovators are faculty who use technology for teaching without CTL assistance. They typically invest under 50 hours of effort into a project.
Findings
This research focused on three major findings dealing with the cost versus impact on teaching and learning when comparing small and large projects:
Value, Satisfaction, and Time Savings Faculty Perceive
Faculty were asked to compare the value they received from completing a technology project with the time they invested (see Figure 1). Of faculty working on large-scale projects, 92 percent felt the value exceeded the effort expended, compared to 71 percent of faculty working on small-scale projects.
Click image for larger view.
Ratings for independent study (IS in the figure; 29 percent) and technology innovators (TI in the figure; 57 percent) in the "other" category were both significantly lower than the CTL large- and small-scale project categories. Interview data helped explain why: The independent study faculty reported that often the course development takes too much time—they do not have enough interaction with their students. We conclude that low rating scores from independent study faculty do not necessarily pertain to developmental project work; rather, they can be attributed to the nature of independent study at BYU.
In addition to faculty perceiving greater value from large-scale projects, their overall satisfaction level was on average about 10 percent higher than faculty who completed small-scale projects. Faculty in the tech training (TT) category were significantly less satisfied (by as much as 60 percent) with the outcomes of their experiences.
In terms of faculty reporting that their projects saved them time, small-scale projects ranked higher than large-scale projects by 15 percent on average. Although small-scale projects ranked higher in productivity, some small-scale projects are designed for pedagogical or strategic purposes and some large-scale projects are designed for tactical purposes. It is also significant to note that Faculty Fellowship participants had the lowest score by far in this category, with only one project out of seven reporting their projects saved them time (see Figure 1). Although this percentage is quite low, the primary purpose for such projects was not to increase productivity but rather to enable Faculty Fellows to teach concepts that they could not teach before, or to enhance aspects of their teaching material.
The Frequency with Which Faculty Use Their Projects
Another measure of a project's value is how regularly faculty use it in their classes. Some faculty projects were designed to be used once or twice throughout a semester, while others were designed to be used nearly every day. For example, the Virtual Audiometer, a simulation designed to allow students to practice giving hearing tests to "virtual patients," is used nearly every day in audiology and speech language pathology classrooms. Another faculty member wanted a way to help students understand how a microphone works, so an animation was created to show sound waves and how they are affected by the microphone's diaphragm. The animation is shown only once or twice each semester, and the students grasp the concepts quickly.
Faculty who completed large-scale projects had a tendency to use their projects more frequently than faculty who completed small-scale projects (see Figure 2). On average, large-scale projects were used daily or weekly 69 percent of the time, while small-scale projects were used daily or weekly only 24 percent of the time. Large-scale projects were typically an integral part of each faculty's course, whereas small-scale projects typically were used between one and several times throughout the semester to demonstrate a key point or a section of a lesson.
Click image for larger view.
Another finding of interest is the large number of projects never used. One of seven Faculty Fellows reported that they never used their projects, and an average of 18 percent of faculty who completed small-scale projects reported that they never used the product or service provided by the CTL.
Evaluating the Impact of Projects on Student Learning
One of the survey questions addressed how faculty measured the impact of their projects on student learning (see Figure 3). Faculty responded to the statement "I know that my technology project has had an impact on my students because…" by selecting one or more of the options provided. Seven options were grouped into the following three categories for reporting:
- Direct measures—some measurable form of data, or obvious improvement in test scores or grades4
- Indirect measures—informal student comments or student evaluations, or faculty perceptions of impact on students
- No measures—no form of evaluation conducted, or faculty uncertain about impact
Click image for larger view.
The majority of faculty reported using technology for pedagogical purposes, but more than one-third of the faculty engaged in small-scale projects have not conducted any form of evaluation on their projects or technology use. BYU does evaluations on each course every semester, but not specifically on technology use.
Although 74 percent of faculty used the projects they completed, many did not see direct evaluation as a high priority. Figure 3 shows that faculty predominantly employ indirect measures to assess the impact of their projects (across all categories of projects except TLL). Faculty who completed small-scale projects were less likely (35 percent) than faculty who completed large-scale projects to conduct any form of evaluation or were unsure about the impact of their projects on student learning. In comparison, none of the faculty who completed a large-scale project said that they were unsure whether their technology project had any impact on their students. Technology innovators were 17 percent more likely than any other group to report that they had not conducted any evaluation or were unsure about the impact of their projects.
These findings will help BYU administrators better understand how to facilitate the type of technology projects faculty should use to achieve the greatest impact on student learning. The findings also support considering VOI when determining resource allocation and technology support.
Discussion
Overall, faculty who completed either large- or small-scale technology projects reported positive experiences. The fact that large-scale projects were viewed by participants as having a higher value-to-cost ratio doesn't undermine the importance of the small-scale projects, which were also viewed favorably by faculty. In essence, faculty perception of value increased when they felt they received a proportionally larger output (greater benefit) for their investment of time.
The small-scale projects tended to provide more time savings for faculty, which might represent a focus on productivity over pedagogy.5 Faculty should not dismiss the idea of small projects—multiple faculty could have useful small-scale projects created for the same amount of time and money needed to benefit one or two faculty who create large-scale projects. The decision should be based on faculty needs and the available funds.
Interestingly, 100 percent of the technology innovators reported using their projects at least weekly. Instructional design consultants at the CTL selected these individuals from faculty they considered innovative users of technology. Faculty Fellows and tech trainers were the only other categories that came close to such frequent use of their projects, reporting 85 and 80 percent use on a daily or weekly basis, respectively.
After receiving the data from the surveys, we were pleasantly surprised—and concerned—to find the technology innovators using their projects so much more frequently than everyone else. In talking with these individuals, we learned that some of them were using the latest technologies for teaching and learning in their classrooms. Of the other faculty, most considered themselves to be technology innovators if they used PowerPoint, Excel spreadsheets, Blackboard, e-mail, or video clips to increase productivity.
Many of the faculty who completed small- and large-scale projects at the CTL also used common technologies just as frequently in their classrooms; however, they were interviewed primarily on their technology projects rather than their use of additional technologies. Conversely, since the technology innovators were not working on technology projects, they were asked the same questions as the rest of the respondents. Instead of applying their experience to a project, they were asked to apply their experience to their general use of technology, which included low-threshold technologies such as PowerPoint and Excel. Therefore, the frequency with which they used these low-threshold technologies might be approximately the same as the rest of the faculty, although the frequency with which faculty use their technology projects is less. This is speculation, of course, as the faculty who completed large-scale projects were not interviewed on this aspect of technology use.
The technology innovators reported that they used technology primarily to increase productivity in their classrooms. They used Blackboard as an integral part of each class, for example, to store information and grant students access to content including syllabi, course documents, and grades. One technology innovator commented:
The frequency with which faculty used their projects correlates with their reasons for using technology. From the interviews, the most common responses faculty gave for using their projects can be grouped into four categories, as shown in Table 1. The table also provides examples of how faculty use technology in each of the categories.
| Table 1 | |
| Reasons to Use Technology | |
| Reasons to Use Technology | Examples of Technology Use |
| Pedagogy/Learning | Faculty use technology to teach new concepts or skills, enhance learning, elicit discussion, and improve test scores. |
| Efficiency/Productivity | Technology helps faculty pace themselves with their lectures, become organized, and free up time for themselves or students. |
| Access/Convenience | Faculty use technology to provide students with material they can use outside the classroom, enable students to receive material if they missed class, and store classroom material for student access on demand. |
| Sociality/Connectedness | Technology enables faculty to connect with students, gain prestige, recruit students to their programs, and meet requests from the administration to use technology. |
Of the four reasons listed in Table 1 for faculty to use technology, pedagogy/learning was the most common motive mentioned. Although the technology innovators primarily used technology to increase productivity, they only account for 23 percent of the faculty interviewed. During the interviews and in the surveys, faculty often commented that they used their projects to improve pedagogy rather than for purposes of productivity and convenience. Furthermore, every faculty member who completed a CIMA project reported that using their projects allowed them to teach concepts that they could not teach before.
One faculty member who completed a CIMA project worked with personnel at the CTL to create a simulation called Mammalian Neurons. The simulation allowed students to perform intricate physiology experiments that would not have been possible without the use of very expensive equipment. The faculty member who worked on this project noted:
Along with using technology to teach new concepts, faculty also reported they frequently used technology for additional pedagogical reasons, including to enhance learning. In fact, 63 percent of the Faculty Fellows reported that their primary purpose for using their technology projects was to enhance their material, not to increase productivity.
One Faculty Fellow worked with the CTL to create interactive video simulations to help her students gain a greater understanding of why businesses operate the way they do throughout the world. She explained:
Of the faculty who completed either a large- or small-scale project, 26 percent said they never used their projects or the training they received from a tech training session. Responses from the qualitative data provide four primary reasons why faculty never used their projects, as follows:
- Faculty received a new teaching assignment
- Project contained errors
- Faculty feared receiving low student evaluations
- Project was not complete
Two of these responses (errors in the project and incomplete projects) relate to work performed by the CTL. Three of the four responses faculty provided for not using their projects were beyond their control. We consider each in turn.
New Teaching Assignment
Many faculty were enthusiastic about working with the CTL on a project. However, a common topic of conversation for these faculty was that before or shortly after the project was complete, they received new teaching assignments. For example, one person who completed a Faculty Fellowship received an assignment to teach a different class as soon as his project was complete. He never used the materials created for the original class. When asked if the faculty member assigned in his place was using the materials, he responded as follows:
Errors in Project
Another common reason faculty did not use their projects was small errors or glitches. Some errors were not discovered until the faculty member attempted to use the project in class. One faculty member shared her experience with errors in her project:
In the majority of instances in which faculty could not use their projects for a period of time, CTL personnel knew of the glitches and began fixing the errors to enable faculty and students to use the projects.
Fear of Low Student Evaluations
One faculty member who completed a Faculty Fellowship had a compelling reason never to use the project that she completed with the CTL—fear of receiving low student evaluations:
Incomplete Project
According to the interviews, faculty have high regard for full-time personnel at the CTL. Faculty often mentioned how nice it was to work with such a dedicated, creative, and hard-working group of individuals. The problem with lack of follow-up seems to stem from projects handled by student employees at the CTL.
Students graduate, move away, or accept a new position. Occasionally, the faculty working with these students are not assigned a new student to assist them with their projects. Some faculty seem to fall through the cracks with their projects in such cases.
One faculty member who lost contact with the student employee assigned to him explained:
Conclusion
Overall, faculty who completed large-scale projects perceived greater value and satisfaction from using their projects, used them more frequently, and were more likely to conduct a direct or indirect form of evaluation than faculty who completed small-scale projects. Much of the cost for the large-scale projects was shouldered by the institution, facilitating projects that in many cases never could have been done by faculty alone. In addition, faculty were more aware of the impact of their projects than faculty who created small-scale projects, although small-scale projects provided a variety of benefits to faculty as well. See Table 2 for a comparison of the overall significance of large- and small-scale projects. Percentages in the table correspond with the faculty survey data.
| Table 2 | |||||
| Significance of Large- and Small-Scale Projects | |||||
| Project Size | Overall Value | Overall Satisfaction | Frequency of Use | Time Savings | Evaluation* |
| Large Scale | 92% | 92% | 69% | 38% | 100% |
| Small Scale | 71% | 82% | 24% | 53% | 65% |
| * Awareness of project impact | |||||
Only in the category of time savings did faculty who completed small-scale projects have higher rankings than faculty who completed large-scale projects. These results suggest that many of the faculty who complete small-scale projects use them primarily to improve productivity and efficiency in their classrooms. Nonetheless, some small-scale projects aim to improve pedagogy, accessibility, and communication with students.
A major finding is that a minority of project implementers collect direct measures of the impact on student learning. University organizations could facilitate direct measurement by making evaluation of each project's implementation and impact on learners a core practice and by helping faculty plan and implement formal evaluations as part of their projects.
Far too many projects completed at the CTL were never used in the classroom (see Figure 2). University and instructional design personnel might want to research whether the projects created are actually being used and, if not, whether they are incomplete or unusable because of errors.
Future research should focus on how soon products are implemented after completion and how long they remain in use. Since a large number of projects were never implemented because of student employee issues, future research could target managing student programmers to ensure reliable software development and that all projects produced are used.
Research could also focus on specific aspects of large-scale projects that make them so successful. A repeat of this survey in 6 to 12 months might prove beneficial in looking for similar trends in technology use as well as to assess areas of improvement.
We offer the following recommendations:
- Remember that evaluation is important although often neglected. Make sure technological projects of all sizes are evaluated. Ideally, every project should have an element of evaluation incorporated into the process.
- Although large-scale projects seemed to yield the greatest value in this study, both large- and small-scale projects should be considered. Determine whether funds should be allocated toward large-scale projects, which can yield great value for a few faculty, or to a greater number of faculty to complete small-scale projects, even if the perceived value and satisfaction are not as high.
- VOI should be considered in addition to ROI with faculty projects and technology use.
In general, organizations should first determine the degree of evaluation performed among faculty who complete technology projects. The resulting information could inform implementing or improving existing evaluation plans and influence future project planning and funding by demonstrating the VOI of technology projects for teaching and learning.