During 2003, the North Dakota University System began to be concerned about the cost of supporting multiple course management systems. Since 1997, the 11 NDUS institutions had used 9 different course management packages,1 including one homegrown product (HTMLeZ) and such proprietary products as Blackboard, WebCT, and e-College. The University of North Dakota (UND), for example, uses both Blackboard and HTMLeZ.
During the fall of 2003, a system-wide committee including faculty, technical staff, and CIOs from all 11 campuses was formed to evaluate moving the North Dakota University System (NDUS) to one CMS. While this effort went forward, staff at UND’s Center for Instructional and Learning Technologies wondered what might happen if the 300 faculty who used Blackboard needed to transfer their courses to a new CMS. This concern generated a search for research from other institutions that were studying a move to another CMS or had done so.
We found several other universities or systems that were considering consolidating on one CMS or transferring existing courses to a new CMS. The State University of New York (SUNY), University of Buffalo, University of Notre Dame, Minnesota State Colleges and Universities System, and University of Wisconsin System had completed studies of multiple CMS products. Recent EDUCAUSE Center of Applied Research (ECAR) research bulletins by Gallagher2 and Hanson and Robson3 are useful guides for selecting a CMS and ensuring it supports the institution’s future goals and needs. In addition, several evaluation tools help institutions or systems choose a CMS, including the Committee on Institutional Cooperation’s guidelines,4 the Lguide,5 and EduTools.6 The interactive, Web-based EduTools site helps institutions compare more than 65 CMS products—some open source and others proprietary—on 42 different functions or characteristics ranging from whether the product offers chat functions, to testing, to compliance with disability standards. Neither EduTools nor other sources had information on whether a course in one CMS could be converted to another CMS or how complete and accurate that conversion might be. We were unable to determine whether and to what extent a course in Blackboard could successfully be converted to another CMS.
This lack of research on how well courses converted from one CMS to another was surprising, largely because other institutions were clearly contemplating or implementing a change in CMS. The NDUS was not the only system facing constrained budgets and the increasing cost of CMS licenses and supporting multiple products. In 2003, 56 percent of public universities experienced a decrease in their academic computing budgets,7 which prompted a reexamination of CMS costs on many campuses. According to results of the 2003 EDUCAUSE Current Issues Survey, IT funding was the number one issue EDUCAUSE members felt must be resolved for institutional success, and course management systems were ranked sixth in the category of requiring the most institutional resources.8
Several CMS providers claim to provide accurate transfer of course materials from another CMS, although such claims do not appear to have been tested by conducting a course-to-course conversion. Furthermore, what would such a transfer mean to faculty, in terms of time and effort taken to evaluate the accuracy of their transferred course? After several months of deliberations, the NDUS selected Desire2Learn as the system-wide CMS. Thus, Desire2Learn became our target CMS product, and the research questions became the following:
- Would all course content of a Blackboard course convert to Desire2Learn?
- If course material converted and for those portions of the course that did convert, how intact and accurate would the content be? In other words, was anything lost in the conversion/transition, or was any material garbled or otherwise made unintelligible or inaccurate?
- How would faculty assess the ease of Desire2Learn, and would they be willing to convert their courses and use another system?
Method
Ten faculty in the Academic Affairs Division at UND evaluated the transition of a Blackboard course to Desire2Learn. These faculty represented five of the seven Academic Affairs colleges and schools and the Honors program. All were users of Blackboard, one for one year and several for four years. We chose faculty for this study for two reasons. First, faculty would need to determine what had converted successfully from their earlier course, what had not, and what had but was inaccurate. Second, it was important to ask faculty’s views of the new CMS after they had gained some experience with the new tool.
The Center for Instructional and Learning Technologies contacted Desire2Learn to request assistance with this study. In summer 2004, Desire2Learn transferred a UND Blackboard course into Desire2Learn using their proprietary conversion tool and provided access to their product for each faculty member participating in the study. The course had been created by center staff and was commonly used to train faculty on how to use Blackboard, and it included content in all feature areas of Blackboard. We used this course so that no intellectual property issues would arise as a result of sharing the materials with Desire2Learn.
The faculty were briefed on how to access the Desire2Learn system, issued user names and passwords, and shown where the course was located via a test log in. Faculty assessed the course in their offices or the center’s faculty lab. While technical support was available to troubleshoot problems, the purpose of the test was to have faculty assess the operation of Desire2Learn, so no more assistance was provided than needed to get to the course or the evaluation instrument.
An Instructional Content Screening Instrument (ICSI) was constructed to capture all the features and content in the original Blackboard course. The ICSI helped faculty determine which content of the Blackboard course converted to the Desire2Learn course and, if the material converted, to what extent it appeared to be intact and accurate. For this study, "intact" meant "the content appears whole or unchanged with no missing parts or elements," while "accurate" meant "the content appears to be precise and free from error."
After completing the ICSI, faculty were asked about their (admittedly modest) experience with Desire2Learn, including questions about its ease of navigation, its effect on teaching, the accuracy and completeness of the conversion process, their willingness to move to a new CMS, and their assessment of students’ willingness to move to a new CMS. Given the small number of participants, responses are reported by frequency and percent only. Because this study may be the first of its kind and because it used few faculty, it is at best a preliminary exploration to be followed by studies that include more subjects and delve into these issues more deeply. Despite these qualifications, the study produced some interesting data and intriguing questions for further study.
Results and Lessons Learned
Figure 1 presents the data on which Blackboard content converted and which did not. Content in the Blackboard course that did convert were Web resources (90 percent of faculty thought this was converted), assignments (80 percent), course documents (75.17 percent), and the syllabus (100 percent). Parts of the Blackboard course that did not convert were the gradebook (86.67 percent of faculty found this did not convert), control panel (74.44 percent), library connections (100 percent), communications (97.89 percent), tests and surveys (95 percent), faculty name (100 percent), and announcements (80 percent). Summarizing across all parts of the course, faculty indicated that 46.91 percent of the course content converted and 53.09 percent of the course content did not convert. This might lead one to a "glass half empty or half full" conclusion, but in fact the parts of the course that did not convert are often time-consuming to reconstruct (for example, test surveys and the gradebook).
Click image for larger view.
Concerning the course content that converted, faculty were asked if it converted "intact." A total of 70 percent of faculty reported that content converted "intact," while 5 percent reported the content converted "partially," and 24 percent reported content did not convert "intact." When asked if the converted material seemed to be "accurate," 80 percent of faculty indicated that the content appeared to be accurate, 5 percent indicated the conversions were partially accurate, and 15 percent indicated the conversions were not accurate. This means that faculty—who would need to restore course content that did not convert intact and accurately—will face increased workloads and frustration should an institution choose to change from one CMS to another.
However, it is important to contrast the faculty answers to the next five questions to these findings. When asked if they felt the Desire2Learn interface was familiar or easy to navigate,
- 20 percent of the faculty indicated the design was familiar/easy,
- 50 percent felt the interface design was moderately familiar/easy, and
- 30 percent indicated it was minimally familiar/easy.
In answer to the question on how changing to the new CMS might affect teaching,
- 50 percent of the faculty viewed the CMS as a tool, having no effect on their teaching,
- 10 percent felt the CMS might improve their teaching,
- 10 percent felt the switch might lose work, and
- 30 percent felt it would be too much effort for too little gain.
As for their impression of how well Desire2Learn converted the Blackboard course,
- 40 percent of the faculty reported it was not intact and somewhat accurate,
- 20 percent reported it was moderately accurate and intact,
- 20 percent reported it was somewhat accurate and intact, and
- 20 percent reported the conversion not acceptable.
Surprisingly, when asked to indicate their willingness to change to another CMS, 80 percent of the faculty reported a willingness to change, while 20 percent were not willing to change or opposed to it (see Figure 2). This is intriguing, since it captures the willingness of faculty to switch to another CMS despite the work of ensuring materials in the new CMS are intact and accurate. And, these faculty had already seen how well (or poorly) the conversion process worked, so their willingness to change is doubly interesting.
Click image for larger view.
Faculty were also asked how willing they thought students might be to change to a new CMS, and their answers were again largely positive: 70 percent of faculty felt the students would be willing, and 30 percent felt students would be unwilling or opposed to the change. In other words, despite some negative comments, the faculty seem to be open to a change in CMS.
What lessons can we learn from this exploratory study?
- First, institutions and systems seeking to save money by consolidating with one CMS need to factor in the time and effort of faculty who will need to review, correct, and revise content once the course has been converted. Since this preliminary examination revealed half of the course might not convert to the new CMS, the decision by institutional leaders to move to a new CMS represents a real cost to the faculty member, department, and college.
- Second, institutions and systems need to investigate and evaluate the conversion tools of potential CMS products.
- Third, CMS providers need to improve and market their ability to convert material from one CMS to their own product, since this could well be an important selling point to institutions already invested in a particular product.
- Fourth, these considerations also may apply to the new, emerging world of free or open source CMS products.
- Fifth, despite the work of fixing partially converted courses, faculty were not averse to trying a new product, and they felt students would be willing to change too.
Faculty may not be the brakes to change that some make them out to be, but they are legitimately concerned about the time and effort needed to make the transition a success for their courses and students.