Design and Development of a Faculty Technology Practices Directory

As one part of a quality enhancement plan, North Carolina State University recently implemented a technology initiative with an initial focus on evaluating and improving classroom technology, piloting technology-rich workspaces for student projects, and initiating an internal grants program for faculty.¹ An advisory committee directs the initiative with stakeholders from the faculty, distance learning and information technology offices, faculty teaching/learning center, library, and university assessment.

The advisory committee began a conversation in the spring of 2006 on evaluating faculty uses of technology. They set a goal of gathering systematic and broad information regarding current uses of technology as it relates to pedagogy (instructional techniques). In response to this discussion, the committee funded the design and development of a searchable Technology Practices Directory (TPD) to study how faculty use technology to impact learning. The term "directory" is purposeful, as it suggests a tool faculty can use to find others with similar technology interests in the spirit of forming productive partnerships and communities. The advisory committee specifically avoided calling the project a "survey" or "database," which might connote one-time data collection rather than a dynamic and changing information base.

The directory is designed to serve the needs of multiple stakeholders, consistent with the diversity of the advisory committee. The primary purpose is to help university assessment staff research existing technology practices in campus-based and distance classes. A secondary purpose is to help faculty collaborate with peers by using search features to quickly locate colleagues using particular technologies. The directory could also help faculty document and promote innovations in their teaching through a public, searchable interface. Faculty can export and print their directory entries for inclusion in a dossier or other documentation associated with reappointment, promotion, and tenure decisions.

While the directory is designed primarily to benefit university assessment and faculty collaborations, faculty-serving organizations could also use the data for planning. A faculty center for teaching and learning could use the directory to identify popular tools and form learning communities, for example, or identify peer mentors and tool experts to lead specialized professional development. Information technology and distance learning groups could use the directory to identify underutilized tools for which additional training might be required or heavily used tools that might indicate a need to shift resource allocation.

The directory project is somewhat unique in its focus on documenting faculty uses of technology in association with learning activities at the campus level. Other faculty directories publicize research and areas of expertise nationally in a public profile, but teaching practices at the campus level are not emphasized.² Peer review of teaching systems enables faculty to document teaching practice with student work samples and receive collaborative feedback from reviewers.³ The TPD is more specialized, however, in assessing teaching with technology specifically and fostering non-evaluative collaborations on campus. It does not include a feature for faculty to comment on or rate the quality of their peers' technology uses.

TPD design and development was led by a core design team consisting of three staff from university assessment familiar with survey design and a faculty member in the Instructional Technology Program (the author) who received a grant from the technology initiative to advise the directory's theoretical design and process results. The design team received its charge from and reported updates to the advisory committee. The committee provided funding and in-kind support, assisted with focus groups, reviewed directory drafts, and ultimately approved continuation of the project based on progress.

The directory project has progressed in phases, beginning with the theoretical design for what information to capture and how to associate technology information with pedagogy and learning. After finalizing the theoretical design, the design team proceeded with the interface design using both print and Web-based prototypes. Finally, the design team prepared a phased release and marketing plan to encourage faculty participation. This article outlines challenges, proposed solutions, and lessons learned in these three phases. It concludes with faculty reactions to the directory and proposed enhancements that might improve participation.

Theoretical Design

A key challenge for the directory project was to help faculty associate their technology practices with student learning. While most faculty can tell you about a tool they use in their course, it is more difficult for them to describe how or why that tool actually affects learning. Thus, a key goal for the directory was to guide faculty in making informative technology-pedagogy alignments that could be summarized to depict the impact of technology on learning.

Based on the assumption that most faculty can easily list the tools they use in a course, we sought a taxonomy of tools that most faculty could immediately understand (that is, "Does the tool you are reporting fit in any of these categories?"). We needed a taxonomy that would not just list tools but also capture information on pedagogy, either explicitly or implicitly. After researching numerous classifications of both tools and strategies,⁴ we identified two tool taxonomies that directly addressed our needs—the Media for Inquiry, Communication, Construction, and Expression taxonomy⁵ and the taxonomy of cognitive tools used in support of open-ended, student-centered learning environments.⁶ After eliminating duplicates and grouping some items, we collapsed these two taxonomies into one set of tool-supported activities. Table 1 shows 10 common instructional activities and example tools that support them. The difference between "information" and "representations" in the activities column is one between traditional, text-based information sources and new media representations of information such as audio, video, or simulated worlds.

The adapted taxonomy provides for explicit pedagogy by suggesting 10 common activities to which most tools could be applied. For example, someone listing a Web annotation tool in the directory might align their tool with the activity, "integrating something new with existing information." The taxonomy also helps inform instructional practices implicitly. Implicit across the 10 activities are different groupings of items that allow the design team to make assumptions about the quality of teaching/learning with technology on campus. If faculty report that they use tools requiring student analysis or integration, this provides evidence of information processing in classes. If faculty report tools requiring students to plan goals and create new ideas, this provides evidence of faculty adopting project-oriented or constructivist activities. Three categories of communication and collaboration tools in the taxonomy would provide evidence of instructor-centered teaching (that is, one-way communicating) versus student-centered, collaborative learning (two-way communicating, collaborating).

Another critical variable considered in the theoretical design was who actually uses a tool. Faculty could use tools to conduct most of the 10 activities as they prepare course materials or teach/lecture. Conversely, students could use tools to conduct most of the activities as part of class assignments. In terms of impact on learning, we would hope to see significant evidence of students using tools to learn actively as opposed to just faculty using tools to teach, lecture, and deliver content. Thus, the directory needs to capture this distinction.

Research Focus

Based on this theoretical design with 10 activities, the TPD addresses the following research questions:

• Which of the 10 activities are applied most and least frequently by faculty and, conversely, by students? What do these activities suggest with regard to general pedagogy or student learning?
• Which tools are used most across the faculty? What do these uses indicate with regard to general pedagogy or student learning?

Small Group Evaluation

To assist in revising the theoretical design, the design team developed several drafts of a data collection tool in print form. We shared these mock-ups with the advisory committee both informally through regular meetings and formally through a focus group where 10 committee members completed the forms provided in the tool and provided comments.

The committee suggested the data collection tool begin with a list of general tool categories (such as office tools, course management systems, communication tools, and digital audio/video tools). Faculty would select a category and then write in the name of a specific tool or feature (Excel, discussion board). This approach would aid end-user searching because someone unfamiliar with specific tools or features could search a category to retrieve associated tools and features. Under the digital/audio video category, for example, a search might indicate that faculty use iMovie, Windows Movie Maker, and Camtasia.

The committee expressed mixed reactions to the 10 activities during review. Some faculty quickly aligned their tools with the activities, while others commented that the activities were more appropriate for information technologies used in the social sciences. Some suggested that faculty using machine or non-information technologies in the hard sciences (nuclear magnetic resonance imaging, virtual microscopy, computer-controlled knitting machinery) might have trouble aligning their tools with activities such as "analyzing or manipulating information/representations." The wording of "information/representations" did not correspond exactly to the real objects (for example, cells) with which machine technologies interact. This issue was tabled until after the general release of the TPD.

After the general release, a few faculty again expressed difficulty aligning their tools with the activities. In response, the design team has changed the wording under certain activities to "information, representations, and physical artifacts" so that the directory will better solicit both information and non-information technologies. Also, tool examples from faculty in scientific disciplines were added to diversify Table 1.

Interface Design

The distance education office provided in-kind support to the directory project in the form of two Web applications programmers, who consulted with the design team to build the data collection tool. The programmers took the theoretical framework and translated it into a set of Web-based PHP forms that save entered data to a database. This lengthy process took place over eight to nine separate meetings in the fall of 2006, with time between meetings for the programmers to address requested changes from the design team.

In all, five sections or forms are included in the directory data collection component. Faculty who visit the directory for the first time begin by completing Section I, Contact Information. This page requires faculty to provide first and last name and select their title, college, and department from pull-down lists. Faculty also have the option of entering a campus address, e-mail address, phone number, and personal Web site URL.

After entering contact information, the TPD directs faculty to Section II, Course Information. Requested information includes course prefix and number, college and department, any cross-listed college or department, primary level of students who take the course, approximate number of students who take the course, and teaching method for the course (face-to-face, online, blended, other distance method).

Section III, Technology Information, prompts faculty to think about a single technology or tool they use in the course. They are reminded they can return to this page multiple times to enter information on additional technologies or tools. First, faculty must select a general tool category to which their first reported tool applies. The general tool categories include course management systems, Web page editors, digital audio/video or graphics, Internet/online resources, modeling software/simulations, GIS/GPS, office software, statistical/analytical software, programming software, electronic communication/collaboration, classroom presentation, and other. Second, faculty provide the name of the specific tool. For example, under the general category "office software," faculty might enter "Excel." Finally, faculty see the full list of 10 activities and are asked to mark all for which they or their students use the reported tool. If faculty report Excel, for example, they would probably align it with an activity such as "analyzing or manipulating information/representations/physical artifacts."

As shown in Figure 1, faculty can receive pop-up information on the Section III Web form that prompts them to align their reported tool with different activities. The pop-ups—which display when the user rolls a mouse over the hot text "What's this?"—help faculty understand the types of tools that generally apply to an activity. After they submit this page, the reported tool can be associated with a general tool category and learning activities.

Click image for larger view.

The form for Section IV, Details of Activities, is dynamic and built entirely from the activities faculty select in Section III. Continuing with the example from Section III, if faculty report that the activity "analyzing or manipulating information/representations/physical artifacts" was associated with Excel in their course, Section IV would prompt them to add further details about the "analyzing" activity.

Specifically, three elaborations are requested. First, faculty must indicate who uses the tool to "analyze"—the instructor, the student, or both. This is an important field for follow-up searching and research purposes, to determine if faculty mainly use tools to teach and deliver content or if they also involve their students in using tools to process and learn content. Second, faculty must report how important the tool was for the specified activity—critical, important, nonessential, or detrimental. For example, faculty might report Excel is "important" in helping students analyze information. Figure 2 shows part of the Section IV Web form that prompts faculty to indicate who uses a tool for a designated activity and the tool's importance for that activity. Different campus agencies requested this information to determine the overall value of tools they provide and support. Third, faculty can use an open-ended text-entry box to give an example of how they or their students use the tool for the specified activity. Figure 3 shows part of the Section IV Web form that prompts faculty for this information. After faculty submit this page, the reported tool can be associated with a general tool category, activities, users, an estimate of value, and various descriptions of use.

Click image for larger view.

Click image for larger view.

The data collection tool ends with Section V, Infrastructure. Faculty are asked to check all that apply from a list of infrastructure items needed to support their reported tool (access to Internet in classroom, access to Internet at home for distance education, computer labs, and so forth). An open-ended text-entry box requests recommendations for infrastructure improvements that would optimize faculty use of the reported tool. After faculty submit this page, the reported tool can be associated with a general tool category, activities, users, an estimate of value, various descriptions of use, and basic infrastructure items necessary to support the tool.

The Infrastructure form ends with two final selections. Faculty must elect whether their data may be displayed and made searchable in the public directory, and they must select where to go next: submit and exit, submit and provide information on another tool used in the currently active course (which takes the user back to Section III), or submit and provide information on another tool in a different course (which takes the user back to Section II).

Once faculty have added courses and tools to the directory, they can log in from the directory entry page and see a summary page of courses and tools associated with their campus ID (see Figure 4). Faculty can choose "edit contact information," which takes them to Section I; "add a new course," which takes them to Section II; "edit course information," which allows them to edit courses already created in Section II; "add a new technology" for a listed course, which takes them to Section III; "edit technology," which allows them to edit technologies already added for a listed course in Section III; or "remove," which allows them to delete a tool entry.

Click image for larger view.

The directory includes a search page where users may browse technologies shared for various courses. Two open-ended search fields are provided to search for keywords or instructor name. Otherwise, users employ pull-down menus to search for tools associated with one of the 10 activities, one of the general tool categories, a specific user group (instructor versus student-oriented tools), a specific college, or a specific department (see Figure 5). Users can export the search results to a .csv file. Currently, the directory search is restricted to campus users with an active ID, to prevent data mining from external commercial interests who might harass faculty. All faculty, staff, and students have full access.

Click image for larger view.

Marketing and Release

A phased release was planned for the data collection tool. Faculty who had received internal grants from the advisory committee or who were on the advisory committee were asked to complete tool entries for a course or two in fall 2006. We anticipated many in the general faculty would want to search and browse a few existing tool entries before diving in to share their own, so we planned to populate the directory with examples ahead of the general release.

After the directory was populated with initial data, we enlisted the aid of the provost's office to introduce the directory by e-mail to all faculty on campus. A printed postcard followed, describing the directory's purposes and inviting faculty to participate. The invitations included several selling points, including the ability to form collaborations and document innovative teaching. Links to the directory were posted on the main technology initiative page under the auspices of the advisory committee, and the design team prepared a short article for a campus newsletter.

Preliminary Findings

The first round of submissions has progressed slowly. In a period of eight months (through May 2007), 89 faculty of 2,000 have visited and entered data, with only 43 completing a full entry. The directory includes complete information on 61 tools, but 23 of those entries came from 5 faculty, with the remaining 38 entries entered by 38 faculty (that is, just one entry each). Thus, most faculty in the directory reported one tool and stopped.

We suspect most faculty use 8 to 14 technologies in teaching their courses, but to date they have been unwilling to enter this data in the directory. Further, even though the directory prompts faculty to report one tool at a time and repeat tool entries for each course taught, several faculty lumped multiple tools and activities in one entry, hinting at their desire to finish this task quickly. Finally, many of those completing full entries are associated with the advisory committee, not drawn from the general faculty. As noted, committee members were asked to try out the directory first and enter some information from their courses. If not for this prompted response, the directory would contain little to no data.

The directory response rate is obviously lower and from a more narrow population than desired. Despite its shortcomings, however, reported data include a diverse range of activities and tools, meeting the goal of the advisory committee in capturing broad information about technologies used on campus. Further, 54 of 61 tools reported were used by students in some capacity, with only 7 tools used exclusively by faculty to lecture or teach in a traditional mode. Many technologies are clearly used as part of student-centered learning activities, not just for faculty delivery of information.

In terms of activities, faculty in the directory population were most likely to use technology for two-way communication, creating information, and assessing students (see Table 2). In terms of tools, faculty reported using classroom presentation tools most often, followed by course management systems and other Internet tools and resources (see Table 3). At least three or more faculty reported using tools in the categories of electronic communication, digital audio/video, spreadsheets/databases/word processing, and modeling/simulations. At least two faculty reported using survey, programming, statistical, and Web-page editing software (not included in Table 3).

One-to-One Faculty Reactions

To determine why the directory project started so slowly, the design team contacted several faculty who completed at least one tool entry and other faculty who created an account using the Section I form but went no further. These individuals provided insight into difficulties faculty faced when starting a new directory account. Some problems we had anticipated, while others were unexpected.

At least some faculty confirmed initial warnings by the focus group that the 10 activities might be difficult to translate. As noted previously, the taxonomy was revised following these interviews and the first round of data collection to address this problem, adding examples of technologies that work with "physical artifacts" in addition to "information/representations" to better support faculty in the hard sciences.

One individual suggested the directory's focus on the course level is inappropriate, as he had developed several learning objects as part of research activities that could be used in teaching. He did not wish to associate these with any course, but had to make up a mock course to enter information about his software.

On a related issue, some faculty suggested the entry forms were too repetitive. If they use the same tool for the same purpose/activity across multiple courses, they would like to enter relevant information only once (tool category, activities, users, and so forth) and associate the tool with all relevant courses. Currently, faculty must report the same tool once for each course, since they may use the same tool differently in different courses.

Some faculty indicated the forms were too lengthy and time consuming. An instructor who had developed more than 50 learning objects indicated he would not be willing to "endure" going through Sections III–V of the data collection tool 50 times. Another user recommended we provide a time estimate for completing forms I–V, since he quit not knowing how much time it would take to go through the reporting process.

In response to this feedback, we revised the directory to include clear headers indicating that a total of five forms would be accessed and to show users where they were in the process. We also cut several course-related questions from Section II and reduced the verbosity of text instructions across all forms.

The issue of time is probably a major barrier to faculty use of the directory, given the well-documented demands on faculty. One suggestion by a stakeholder was to open the directory to collaborators who could enter data on behalf of faculty (teaching assistants, instructional designers familiar with a course redesign project). If faculty lack the time to participate in the directory, perhaps others could be empowered to help them out. The counterargument is that staff might not know enough about a course to correctly align tools with activities and users.

Difficulties faced when launching faculty portals are not unprecedented, as noted by the learning object community. Koppi et al. found the lack of a reward structure for developing learning objects and innovative teaching materials was a key barrier for faculty contributors to a learning object catalog.⁷ To encourage a learning object economy, Liber suggested, a need exists to fund, support, and reward "communities of teachers committed to particular pedagogical approaches," and the demand for objects will emerge from sustaining such groups.⁸ For the directory project to realize its potential might require similar incentives and advanced models of collaboration.

Future Directory Enhancements

The design team believes a critical mass of users will be necessary before the TPD becomes self-sustaining and general faculty are motivated to join. We are considering additional incentives and features to improve the adoption rate, including face-to-face introductions of the directory in appropriate venues, an expanded focus at the university system level, extrinsic motivators through leadership buy-in, and intrinsic motivators through faculty-owned communities.

Marketing the Current Directory More Broadly

In addition to promotions such as e-mail and postcards sent to faculty, marketing the directory could include introducing it at new faculty orientations in the fall or during intensive technology workshops in the summer when faculty focus on their course designs. Time could be set aside at these sessions for faculty to create an account and enter information about at least one course and tool. Workshop leaders could help faculty translate the 10 general activities, correctly associate their tools, and answer individual questions.

If individual campuses lack enough faculty interested in teaching with technology to sustain an interactive directory on their own, it might be possible to create a community tool that bridges campuses in a university system or region. The TPD was recently presented to colleagues at the annual meeting of the UNC Teaching and Learning with Technology Collaborative.⁹ Comments from the audience suggest multiple institutions might be interested in contributing to a shared directory. The critical mass of users that has not emerged on our campus may indeed be found by broadening the reach of the data collection. This approach is counter to the initial assessment purpose of the directory in documenting technology uses on our individual campus, but it could better support the secondary purpose of fostering faculty collaborations.

Faculty Assessments as an Extrinsic Motivator

Another strategy to promote adoption of the directory is to sell its value to leaders who will in turn encourage faculty to participate. The primary purpose for the directory is to enable university administrators, deans, department heads, and technology staff to compile evidence of and assess innovative teaching with technology at appropriate levels. For example, a spreadsheet of innovative teaching with technology in a specific college or department could assist with an upcoming accreditation or program review.

Achieving buy-in from leadership is an important factor the design team realized early on, leading to meetings with college technology directors to introduce the directory and ask for their input. The design team also wrote a letter to inform department heads of the tool's value to them personally and provided sample e-mail text they could send to their own faculty requesting (or requiring) participation in the directory. The advisory committee discouraged delivery of this letter, however, suggesting that directory participation should remain entirely voluntary.

Extrinsic motivators should not be discounted entirely, given that the directory provides a good opportunity for administrators to get behind a project that values good teaching. The reappointment, promotion, and tenure process is often criticized for the lack of value it places on teaching; encouraging administrators to use teaching-focused directories as another data source in evaluating faculty performance could serve to elevate the importance faculty place on documenting teaching practice. If faculty realize administrators will never use teaching-focused data, they have less incentive to innovate or document teaching.

Faculty Communities as an Intrinsic Motivator

A secondary purpose for the directory is to help faculty find and collaborate with others, yet the current system only partially achieves this vision. Early directory users suggested the search output should include faculty e-mail addresses, Web page links, and tool links, making it easier to contact and view the work of others. Without adequate means to collaborate, the TPD serves only the primary purpose of data collection, not the secondary purpose of faculty communities.

Early in the design process, we assumed groups such as the faculty teaching/learning center could harvest names from the directory to organize learning communities around tools or activities of interest. This top-down approach for organizing groups is still possible after many users have joined the directory, but the design team believes bottom-up collaborations initiated by faculty would be even more powerful.

A recent article by Barrett lists no fewer than 22 Web 2.0 social networking tools with features that could support faculty community building.¹⁰ TeachAde, for example, allows instructors to create a free account and join subject or topical groups of interest for communication and sharing of resources.¹¹ Elgg is a related open-source tool through which institutions can set up self-contained communities for educators to develop profiles and share lessons and resources.¹² The tool includes a host of features that could support a faculty working group, including profiles, group-edited blogs and wikis, RSS feeds aggregated around topics of interest to the working group, file repositories, and more.

In place of the current TPD, which is more like a static Web 1.0 directory with few features for collaboration, a modified TPD with Web 2.0 collaborative features could include not only profiles for individual instructors but also profiles for groups interested in studying topics such as learning objects or course management systems. Faculty could associate their individual profiles with groups, displaying their names, profile links, and photos on the group's page along with other members. Group profiles might facilitate a sense of community and provide a visual indication of faculty interest in a topic. Specialized learning object groups might only have five to seven collaborators, while generalized distance learning groups might have hundreds. Group profiles would likely be directed by a faculty host responsible for accepting new member requests and posting virtual meeting dates, interesting links, grant RFPs, or other pertinent content for the group.

Lessons Learned

Many lessons were learned in the process of designing and developing the faculty TPD. First, too much attention went to collecting assessment data for a host of agencies—the project's primary purpose. Without extrinsic motivators to encourage participation in an assessment initiative, the data clearly show that most faculty simply will not take the time to participate. And with a detailed data collection tool, the time required to participate was apparently significant for most users.

Second, not enough attention was paid to system features that would help faculty form collaborations and enhance their ongoing work. Thus, if administrators were not going to use directory data to evaluate faculty performance, and faculty were unable to use directory data to connect in meaningful ways, both extrinsic and intrinsic motivators failed to foster faculty buy-in.

This project made progress in defining and refining theoretical and interface designs to help faculty report uses of technology. These designs can still help faculty generate individual profiles, but the TPD should be redesigned to encourage more faculty participation through group profiles and the ability to connect individual and group profiles. The system should support faculty community building primarily, with the assessment interests of other agencies addressed secondarily. Ultimately, more assessment data may be extracted indirectly from faculty and group profiles in robust communities than from an assessment tool no one is motivated to use. These preliminary findings provide a case other institutions can use as they plan similar directories or portals to enhance faculty collaborations around technology and perhaps feed assessment interests in the process.