Faculty and students increasingly want to integrate technology into teaching and learning, using it for more than just taking notes.1 Computer-based examination of student learning is one such use. Implementing computer exams requires a secure testing environment—one that prevents students from seeking answers by scanning their computer hard drives, instant messaging or e-mailing friends, or browsing the Internet. While campuses can establish a relatively secure testing environment in many ways, the challenge lies in finding a long-term security solution acceptable to both faculty and students. Any such solution must not consume an inordinate amount of resources nor impose too cumbersome a process on participants.
The College of Business and Economics at the University of Idaho conducted a pilot study that used commercially available encryption software called Securexam to deliver computer-based examinations.2 A multi-step implementation procedure was developed, implemented, and then evaluated on the basis of what students viewed as valuable.
Two key aspects in this procedure raised some concern. First, the faculty team conducting the study (the authors of this article) preferred that the encryption software itself not negatively affect students (for example, their score on an examination or their individual psychological response). Second, wewanted to provide meaningful support in the form of help with the technology for students to facilitate their use of the encryption software while not biasing their exam scores.
We based the conceptual links between faculty support activities and the students’ examination performance and psychological attributes on the Technology Acceptance Model (TAM; described later).3 The purpose of this research was not to empirically test the model but to provide the conceptual linkages between practical methods faculty can employ and outcomes associated with students’ use of examination encryption software.
The results of this study show that giving students the opportunity to practice using encryption software before taking an examination reduces their anxiety, improves their attitudes toward the software, and improves their perceptions of their performance on the examination. Faculty encouragement in using the encryption software also positively affected student attitudes toward the software. Furthermore, none of the faculty activities influenced student grades on the examination, implying that these faculty activities do not bias performance on the exam despite their positive effects on student attitudes.
The Research
Using commercially available encryption software, we administered an examination at a common time for two sections of a core business course. Participants were 98 undergraduate, mostly junior-level, business students representing seven majors: accounting, economics, finance, information systems, management and human resources, marketing, and production and operations management.
Using their laptops in class, the students downloaded the encrypted exam from a network server using the encryption software and a password. They then decrypted the exam to display it as a word-processing document and a spreadsheet. Once the examination was decrypted, the software allowed students to input answers only into the word-processing document and spreadsheet. In other words, the encryption software prevented students from exiting the specific documents to access hard drives, the Internet, e-mail, or any other software or files.
Five days after the examination, before students received their exam scores, a questionnaire was administered in class. The questionnaire contained items developed from previous research on the Theory of Reasoned Action4 and the TAM.
The Theory of Reasoned Action has proved to be a useful framework for predicting and explaining human intentions and actions. Davis5 and Davis, Bagozzi, and Warshaw6 developed the TAM by applying the Theory of Reasoned Action to the specific instance of technology acceptance. The TAM is therefore useful for understanding what factors affect the beliefs and attitudes of people who use technology as well as their intentions to use technology in the future.7 The TAM was an appropriate framework for this study in particular because the goals were to (1) identify faculty activities that would support the use of the examination encryption software and (2) understand how these activities affect important student outcomes such as student anxiety, attitude, perceived performance, and grade.
Figure 1 shows the original TAM. As mentioned earlier, the focus of this research was the practical classroom application of faculty activities conducted to positively influence students’ perceptions of using examination encryption software. These faculty activities are represented as the external variables. The groups of these faculty activities were defined as instruction, technology support, practice, faculty encouragement, and environment. The relationships studied are between these external variables and students’ perceived usefulness of the encryption software as represented by their perceived examination performance and grade.
Click image for larger view.
The other relationships of interest are between these external variables and student attitudes toward using the software as represented by their anxiety and expressed attitudes. Due to the practical focus of this research, we ignored the intervening variable of perceived usefulness between the external variables and attitudes toward using the software.
Measures
We measured the constructs in the model using a series of questionnaire items. We developed the perceived performance items using the outcome expectancy measure of Stone and Henry,8 while the perceived usefulness items are those developed by Davis.9 The items measuring attitude used the attitude measure of Agarwal and Prasad,10 and we developed additional items in the same spirit. The anxiety items were based on the work of Culpan.11 Modifications to the items used made them appropriate for the context of our study.
A questionnaire created using these items also included items representing each of the five categories of faculty activities (instruction, technology support, practice, faculty encouragement, and environment) and three of the four outcomes (anxiety, attitude, and perceived performance). Actual performance on the examination was measured by students’ examination scores. All of the items asked students the extent to which they agreed or disagreed with a statement, using a five-point Likert-type scale of strongly disagree, disagree, neutral, agree, or strongly agree.
The questionnaire was pre-tested by 11 undergraduate students already familiar with the examination software. The pre-test resulted in the modification of four items and the elimination of four others. The final set of questionnaire items appears in Table 1.
Click image for larger view.
Results
Sixty students from both sections completed the questionnaire, producing a 61 percent response rate. As with any research based on a survey, non-response bias was a concern. However, when we compared the characteristics of the sample to the same characteristics for the student population in the business college, we found that the sample is qualitatively and quantitatively similar to the college population. The only statistically significant difference is that the average age of the sample is higher than that of the college population—not surprising given that the respondents were mainly juniors and some seniors.
We estimated the relationships by ordinary least squares regression. The results for the four regressions appear in Table 2.12 Of the five categories of faculty activities, those related to providing instruction, technology support, and managing the testing environment were not statistically significant for any of the four student-oriented dependent variables (anxiety, attitude, perceived performance, and grade). Faculty activities related to providing practice and encouragement were statistically significant, however.
Click image for larger view.
Opportunity to Practice
The faculty-controlled activity of practicing had meaningful effects on students’ responses, lowering their anxiety with regard to using the encryption software and improving their attitudes toward the software and their perceived performance on the exam. The practice variable did not have a meaningful impact on students’ grades, however. These results are encouraging because they imply that faculty can perform activities to improve students’ anxiety, attitudes, and perceptions without biasing their grades.
Students had several opportunities to practice with the encryption software in class. In a short practice exercise students retrieved a one-page encrypted exercise, decrypted it, answered two simple questions, explored the look and feel of the software’s interface, encrypted the completed exercise, and posted it as if to be graded. On another day, they went through the same procedure using a previous examination. This let them see the exam format and explore a spreadsheet question. The last practice session simulated an actual examination.
The empirical results suggest that faculty should provide students with practice using encryption software. The downside is finding class time to allocate to three practice sessions. Note, however, that by the third session students were obviously comfortable with the procedure, to the point of becoming bored. Two practice sessions are likely preferable—one session to introduce the software and another to simulate the actual examination experience.
Encouragement
Faculty encouragement significantly affected students’ attitudes and their perceived performance on the examination. These encouraging results indicate that faculty activities can influence students’ attitudes and perceptions of performance without biasing actual performance as measured by grades. A disappointing result was that these activities did not mitigate students’ anxiety regarding the encryption software.
Two types of faculty-encouragement activities took place during the practice sessions. In one, faculty emphasized the advantages of the encryption software over traditional paper-and-pencil examinations, pointing out the benefits of using word-processing tools (grammar- and spell-checker, copy and paste, highlight and delete) while avoiding “writer’s cramp” during a two-hour essay examination. Faculty also pointed out that students use their laptops in class daily, so using laptops in an examination setting would be familiar.
Another tactic to encourage students to accept the encryption software was to explain its similarities to ubiquitous technology such as calculators and cell phones and suggest that laptop-based examinations would become the norm in the next few years. The significant results for this category ensure that the faculty will continue this set of actions in the future.
Instruction Activities
The remaining faculty activities (instruction, technology support, and environment) had no meaningful impacts on the dependent variables of interest. The instruction activities included displaying instructions on using the encryption software, both in front of the classroom and on the examination cover sheet. The detailed instructions described how to load, open, close, and post the completed examination. Additionally, a hard-copy examination was provided for students to use as scratch paper.
For several reasons, faculty thought it necessary to explain the process even after three practice sessions. An instructor encrypts the examination with the software, then gives it to and receives it from students electronically. Several different methods for delivering and receiving an encrypted examination from approximately 100 students were considered, including e-mailing the completed examination as an attachment or exchanging it with memory sticks. The most reliable and efficient method for examination submission was to use the university’s Web-based course organizer, WebCT.13 Students needed standardized instructions on how to perform these activities.
Faculty also provided instructions on the storage/saving location of the encrypted, completed examination. Because operating systems vary in displaying the default directory for downloaded files, faculty wanted to standardize where students saved the examination on their hard drives. This made it easier when students started the examination because the software prompts the user to find the encrypted document. When the student has completed the examination, the software also prompts the user on where on the hard drive to save the finished, re-encrypted examination.
Another set of instructions involved the password required by the software to decrypt the examination. Previous experience in similar situations taught faculty that, due to stress and haste, not all students comprehend the password when it is provided verbally at the beginning of the exam—they prefer it written down.
The instruction category of faculty activity shows no significant impact in the empirical tests, yet we believe it is intuitively related to the dependent variables. We suspect its importance is subsumed by activities related to practice and thus will continue the activities in a modified version. While it takes very little effort to display the instructions in front of the classroom, faculty decided to discontinue this activity and provide instructions only on the cover sheet of the hard-copy examination. Casual observation of students completing examinations revealed that students use the hard-copy exam extensively for outlining and calculating initial solutions.
Technology Support
The faculty activities summarized as technology support also had no meaningful effects on students’ anxiety, attitude, perceived performance, or grade. In a positive sense, the results imply that the technology support activities do not bias students’ grades, add to their anxiety, or negatively affect their attitude or performance.
Technology support included having technical support staff in the rooms during the two-hour examination and making available extra power cords and Ethernet cables. Despite the lack of significant results for these actions and the added cost for having technology staff available, faculty will continue these activities until laptop-based examinations become more common in the college, for two reasons. First, only two students consulted the technology staff during the examination, possibly indicating that the majority of students under-appreciated the presence of the staff. The students who needed the help of the technology staff greatly appreciated their presence.
Second, the presence of the technology staff to troubleshoot problematic laptops acted as a deterrent to sabotage. While most students were ready to embrace using laptops in the classroom on a daily basis, a nontrivial subset was openly reluctant to entrust taking examinations to a laptop. To mitigate the instances of self-sabotage during the examination, the faculty stressed ahead of time that only as a last-resort would a student be allowed to take an exam in the traditional way—and only after the technology staff had attempted to fix the problematic laptop. We believe this concern will vanish once the practice of laptop-based examinations becomes more common.
Managing the Testing Environment
The last category of faculty activities related to managing the environment of the examination room. These activities had no meaningful impacts on any of the dependent variables studied. In addition, while these activities did not appear to affect students’ anxiety, attitudes, or perceived performance, they also did not bias students’ grades.
Students were familiar with taking a traditional examination in one large lecture hall. A laptop-based exam, however, required electrical outlets and Ethernet cable connections as back-ups to failing laptop batteries and an overloaded wireless system. Students were assigned to one of three smaller classrooms with these facilities.
A second activity affecting the room environment was assigning students to specific seats to leave an empty seat on either side. This was done to mitigate the possibility of students reading each other’s laptop screens and to provide more desk workspace.
A third action was to have a few faculty members associated with the class roam through the rooms as opposed to having only technology staff in the rooms. Given the lack of significant results for this category of actions, we most likely will not continue using these additional faculty and classroom resources.
Implications for Instruction
The results of this study provide implications for instructors who wish to use encryption software for computer-based examinations. First, providing practice sessions that simulate the students’ completion of the examination using the encryption software helps students become comfortable with the process. These sessions should allow students to simulate decrypting, completing, saving, encrypting, and submitting the examination. Such practice sessions can have desirable impacts on students’ anxiety, attitude, and perceived performance. It also appears that these practice sessions do not influence examination performance as measured by grade.
Second, faculty can positively affect students’ attitudes and perceived examination performance when using encryption software by encouraging them in its use. Encouragement can be in the form of faculty explaining the advantages and inevitability of examination encryption software. Such encouragement does not influence student anxiety about using the encryption software or actual examination performance as measured by grade.
The empirical results also indicate that actions taken to provide additional instructions, technical support, and a supportive testing environment were not considered valuable by the students. In terms of instruction, the non-significant results suggest that faculty need to minimize resources and class time allocated to these activities. However, these activities must not be completely absent. Some efforts are needed to provide instructions, technology support, and a suitable examination environment for students. In other words, faculty must provide the basics in these areas.
The results also show that none of the faculty activities affected actual examination performance. Intuitively, we expected that typing essay examinations with editing tools would produce better results than hand-written ones.14 Nonetheless, the lack of a significant effect of any of these activities on examination scores is arguably positive. While technology can make the process of answering a question easier, in the end, students must understand the concepts and express their understanding through the examination.
Because of the favorable results and valuable lessons learned from our pilot study, we intend to pursue using encryption software for computer-based examinations. Emerging products might better support essay exams versus multiple-choice testing. We are interested in observing how our support activities might evolve as a result of computer-based encrypted exams becoming the norm rather than the exception and look forward to reporting the outcomes in the future.