Conceptualizing the Functional Requirements for a Next-Generation E-Portfolio System

Key Takeaways

• An institution’s administration, faculty, and students all have vested interests in deployment of an e-portfolio system, which can address goals ranging from showcasing student work to assessing student learning.
• When an existing e-portfolio system fails to meet the institution’s goals, the first steps in choosing a new system involve deciding on its desired purpose and required functionality.
• Developing selection criteria that encompass purpose, functionality, and potential barriers to use improve the likelihood that any e-portfolio system meeting those criteria will serve the campus community as intended.

The use of e-portfolio systems in higher education has not grown as anticipated by early adopters when the potential benefits first became apparent, yet the promise remains. The vision of a flexible tool that meets multiple goals — serving as a showcase for student work, promoting self-reflection, supporting assessment and accreditation efforts, moving with students as they left academia — can succeed if each higher education institution plans for the adoption of e-portfolios by starting with the fundamental requirements for its unique community.

Realizing the Vision, Understanding the Reality

The first steps in adopting any information technology consist of deciding what purpose the system will address and what functionality the target institution requires.¹ For an e-portfolio system, these are particularly challenging tasks, given that the e-portfolio community — vendors and practitioners alike — have neither established a one-size-fits-all definition nor a corresponding set of standard functionalities. Higher-education institutions deploy e-portfolios to capitalize on the expected benefits of supporting a variety of student achievements, both as showcases of their work and for self-reflection, as well as providing a more authentic form of learning assessment.² This article aims to share a categorization of functionalities developed over a one-year period by an e-portfolio work group in charge of determining a set of criteria for the selection and evaluation of one institution’s next iteration of an e-portfolio system.

The target institution is a small, urban, liberal arts university that serves approximately 2,100 undergraduate and graduate students with over 50 academic programs in six divisions and schools. The institution has used a home-grown e-portfolio since 2003, the same year a new undergraduate core curriculum and corresponding assessment plans were put in place. The e-portfolio system was designed with a backend database for storage and a web-based interface to provide access for students, faculty, and administrators.

One intended purpose of the e-portfolio system was to assist with implementing assessment plans focused on improving student learning. A second, but not secondary, goal was that the e-portfolio would provide a vehicle for students to better understand their own strengths and weaknesses, as well as provide a place to showcase their achievements for graduate school and job applications. Part of the university’s mission statement is to "engage its community in life-long learning," and the e-portfolio was seen as a way to facilitate this.

All undergraduate students at this institution are assigned to an e-portfolio collection structure based on key elements of the core curriculum. The e-portfolio includes spaces for student work from both first-year and upper-level courses that will demonstrate general student learning outcomes such as writing, quantitative reasoning, speaking, and critical thinking, among others. The system also prompts for less-specific artifacts such as "examples of problem solving skills" or "examples of service and civic responsibility." The core student template is a tool designed to teach students how to think about collecting ongoing evidence of their abilities in order to see their growth over time and to begin to identify their strengths and weaknesses. To facilitate this, students are encouraged but not required to include a brief reflection about each artifact submitted. The students’ academic advisors and, in some courses or programs, their instructors have access to the core or program template portfolios and can provide limited amounts of formative feedback on the submissions. All students also have a personal template in which they can create their own headings, although display options in any of the templates are extremely limited. In all structures, personal or otherwise, the student has the option to make specific items public.

Despite the university’s best intentions, the system originally created turned out to be fairly one-dimensional. It only provided partial assessment support and limited student benefits beyond simple artifact collection, organization, and presentation. In addition to the limited personalization options described above, collating artifacts for assessment was extremely tedious, and the reporting capabilities of the system were primitive and inflexible. Also, and in hindsight, the institution failed to implement some organizational practices (such as engaging users to obtain ongoing feedback) that could have helped improve the existing e-portfolio system (the system was seen as fully completed after deployment). However, following the initial implementation, both e-portfolio and assessment technologies and practices have advanced significantly, and the institution’s own conceptualizations and practices have similarly evolved. The need to move into the next generation of e-portfolio applications became obvious.

The university commissioned a work group of faculty and staff from a variety of disciplines and perspectives to review e-portfolio use and the existing e-portfolio application and to determine a future direction for e-portfolios at the institution. The group was asked to determine the functional requirements of an e-portfolio system without consideration of costs, IT infrastructure, or whether the system would be developed in-house or purchased. The reason for this was to ensure that the group would focus solely on the ideal functional characteristics, consistent with the institutional goals, that the next e-portfolio system should have.

Surveying for E-Portfolio Use

The e-portfolio work group began in spring 2008 by surveying faculty about their perceptions and uses of the e-portfolio in order to inform a needs analysis. Of the 99 full-time faculty members, 58 responded, giving a response rate of 58.59 percent — very high for this type of study.³ This survey, along with subsequent discussions with key university committees, revealed two main priorities for an e-portfolio system:

• Showcase students’ work
• Assess student learning

The results summary of the survey indicated that while the majority of the full-time faculty respondents knew at least something about e-portfolios in general and the university’s e-portfolio specifically, only a small percentage (3 percent) interacted with the university’s e-portfolio system on a regular basis. A large percentage (over two-thirds) had never used it at all. This revealed that although the e-portfolio had been officially adopted and in use for five years, the usage penetration among faculty had reached only 24 percent since its initial deployment (Figure 1). Unfortunately, this phenomenon is not unusual; the research literature on innovation has dubbed it the assimilation gap in the diffusion of innovative technologies.⁴

Figure 1. Faculty Familiarity with Current E-Portfolio System (N = 58)

At the same time, the core curriculum and a fair number of program assessment plans relied heavily on the e-portfolio. For these purposes, students were asked to upload key assignments into their e-portfolios. Some students, particularly Biology majors, have reported that their e-portfolios were helpful when applying for graduate schools or jobs. For example, one student was granted a graduate teaching assistantship based on the strength of her application, which included samples from her e-portfolio. About 87 percent of the university’s traditional students have uploaded at least one item into their e-portfolios, based on data extracted from the system, but the average number of artifacts submitted per student to the core curriculum templates has been only 6 out of a possible 19 core artifacts plus 10 optional items. This rather low number of uploads shows a lower usage penetration among students than desired — another case of assimilation gap in the use of the e-portfolio, this time among students. The extent of e-portfolio use depends also on the specific progam. Nursing, for example, had a 100 percent completion rate for their graduating seniors, but they required use of the e-portfolio at the course level. In this program, all course requirements (including e-portfolio assignments) must be met prior to the final exam. In other programs, such as Intensive Studies, while the use of e-portfolio may be a course requirement, it is merely averaged into the final grade, resulting in a somewhat lower e-portfolio usage. Interestingly, it is possible to observe a correlation between low use of the e-portfolio and low engagement and retention factors, which may also include low grades.

Despite uneven adoption of the e-portfolio, there have been some small victories: Artifacts have been drawn from the active e-portfolios to inform assessments, particularly of the core writing program and critical thinking skills, as well as to inform the curriculum and teaching practices in the Biology department and the School of Nursing. However, the amount of material that has gone into the e-portfolio system to date, small as it is compared to the potential contributions, far outweighs the amount of material pulled out (approximately 200 artifact samples laboriously drawn for review). This seriously hinders the benefits of using an e-portfolio for assessment purposes and constitutes another example of an e-portfolio assimilation gap, this time at the assessment level.

In summary, the university has collected the data to inform teaching and curricular practices, but cannot easily get to it or use it. The students have artifacts that constitute evidence of their abilities, but cannot interact with them in a beneficial way. Most of the faculty have never used the e-portfolio. All this makes clear the need to deploy a more usable e-portfolio system, but the key question is, "What are the functional criteria to consider when planning for the next generation of an e-portfolio system?" Addressing this question was the next task of the e-portfolio work group.

Developing the E-Portfolio Selection Criteria

In the survey, faculty noted some of the barriers to using the university’s current e-portfolio, which include an unfriendly and unattractive interface for both faculty and students. Subsequent conversations with students indicated this is also their main issue with the e-portfolio. Conversely, faculty articulated the desire to have a system in place that would support and showcase existing and potential demonstrations of student growth, as well as continue to support evolving assessment practices. After evaluating barriers to use and positive feedback from the survey, vendor demonstrations,⁵ and results of discussions in the e-portfolio work group, the group drafted the following functional criteria for an e-portfolio system focusing on a showcasing student work and assessing student learning.⁶

1. Tiered Access

The e-portfolio should be accessed and assessed on multiple levels; that is, not all constituents should have access to the same material or the same view of the material. For example, some owner-defined artifacts may be accessible only to the owner, while others may be accessible only by faculty, and so forth. This criterion has two sub-criteria:

1a. External Access: Facilitates the viewing of showcased work by potential employers, family, and friends. Provides different levels of sharing based on artifacts. Access is controlled by the owner of the portfolio, including how long guest accounts are active. Some owner-defined artifacts are accessible only to owners.

1b. Internal Access: Provides for portfolio evaluation and administrative maintenance on multiple levels. For example, access for evaluation might include faculty appraisals, self-appraisals, course appraisals, advising work, or program or university appraisals. System administration access should include individual portfolio auditing, program-level administration, and assignable role-based access. Some owner-defined artifacts are accessible only to owners (for example, course-based appraisals).

As a way to illustrate this criterion, the current e-portfolio allows only the student, the student’s advisor, and the e-portfolio administrator to see the student’s individual e-portfolio. When a student shares an item in the current e-portfolio, everybody who has access to the Internet could potentially see that item if they knew where to look. However, in the new generation of e-portfolios, the tiered access features allow students to control what content can be accessed, by whom, and in what form. These various tiered levels of external security, which the students would assign, provide them with more control over their own work.

2. Matrices of Student Learning Outcomes

Matrices provide a clear visual reference to expected student learning outcomes and levels of growth by course, program, and institution as well as a map to external standards (Nursing, Psychology, Accounting accreditation/certification, etc.). While there should be places in the e-portfolio for the student to self-define structures or presentation formats, overtly articulating the end goals of an education at this institution should provide the student a better opportunity to self-manage learning and see progress. As shown in Figure 2, a degree audit (or checklist) simply tells students which courses they have or have not taken, rather than helping them determine their progress on each of the overall learning goals (matrix style).

Figure 2. Checklist⁷ (left) vs. Matrix (right) Style E-Portfolio Collections

3. Evaluation

Evaluation features provide interactive methods for reviewing, assessing, and encouraging student growth. They can be organized into two sub-criteria based on the type of feedback they provide.

3a. Rubrics for Evaluating Student Learning Outcomes: Rubrics should be interactive and feed assessment data. They should also be descriptive for students, faculty, and administration. Because artifacts may be reviewed at multiple levels (course, program, university), single artifacts should be able to have multiple rubrics attached to them.

3b. Areas for Feedback: Formative feedback at multiple levels is important for both student learning and "stickiness" of the e-portfolio system.⁸ There should be multiple levels and types of feedback by peers, faculty (courses), advisors (letters of recommendation), and possibly assessment team members.

4. Areas for Student Reflection

Reflective learning is one of the university’s core student learning outcomes. The system should provide ample opportunity for a student to reflect on artifacts for self-review and deep learning, promote a student’s ability to make comparisons and connections, and allow a student to set academic, personal, and professional goals.

5. Aggregate Reporting

The ability to collate and view data at multiple levels in aggregate form might include instructor evaluations and progress toward program student learning outcomes, combined with other aggregate program data available to the school or division: external standards, program demographics, and aggregate comparisons. The same functionalities should be available at the core curriculum (institutional) level, along with the ability to do comparisons with retention data and completion/activity statistics.

6. Aesthetics and Usability ("Stickiness")

The next-generation e-portfolio must be perceived as easy to use and aesthetically pleasing so that users feel inclined to use it. Research into technology adoption shows that a perception of easy use is necessary for a new technology to be successfully adopted.⁹ Further, unless it is visually appealing, it will not be a suitable showcase for potential employers.

6a. Easy to Use: This sub-criteria is particularly important for faculty and students, but also for the outside world. An e-portfolio should be easy to view, understand, and navigate; add, create, and change different types of content (Office files, video, graphics, etc); sort, search, and organize (sitemap or file tree); and access by potential employers or external reviewers via a direct URL or link.

6b. Aesthetics: This sub-criteria is particularly important for the outside world, but also for students. The e-portfolio technology should be visually appealing and provide customizable views and looks. This criterion is essential for students to successfully showcase e-portfolio artifacts for potential employers.

7. Other Considerations

Other considerations include the system’s integration with key university applications, particularly with the student information system (in this case, Datatel) and possibly the learning management system (Blackboard); the length of time past graduation or transfer that students and the university have access to student work; and technical criteria (infrastructure, staff, and bandwidth needs).

Weighting the Criteria

After determining and defining the criteria, the next step for the e-portfolio work group was to weight (or prioritize) the criteria. Although several multi-criteria prioritization methods are available, the e-portfolio work group chose the Analytic Hierarchy Process approach, a prioritization method based on pairwise comparison of criteria¹⁰ that is widely used due to its ease of use and accuracy. To simplify the process, group decision support software called Decision Lens was used. The advantage of this software is that it not only automates the AHP calculations but also provides support — such as anonymous electronic voting — to the group decision process. The importance of each criterion relative to the others was discussed by the group, voted upon, and automatically aggregated into the decision support software. At the end of this process, the derived weights for the criteria were calculated by Decision Lens as shown in Figure 3. In this figure, the number next to each criterion constitutes its weight, relative to the whole. For example, Aesthetics and Usability (Stickiness) constitutes 23.4 percent of the total combined weight (of 100 percent) of the criteria. Evaluation has 19.1 percent of the total weight, and so forth.

Figure 3. Criteria and Weights of the "Ideal" E-Portfolio

As can be seen in Figure 3, Aesthetics and Usability, Evaluation, and Tiered Access are consecutively the most important e-portfolio criteria for an e-portfolio focusing on assessment and showcasing student work. Areas for Student Reflection, Aggregate Reporting, and Matrices for Learning Outcomes followed. As mentioned before, these weights were calculated as a result of the pairwise comparison of the criteria by the group.¹¹

It is important to note here that the desired functionalities (and their relative importance) of a new e-portfolio system will vary from institution to institution. Therefore, each specific institution should review the prioritization of the functional criteria for e-portfolio systems planned for their own campuses. This is one of the advantages of using a structured decision-making approach — it provides the ability to develop an e-portfolio selection model that suits the individual needs of various institutions.

Next Steps

Since the initial review of the home-grown e-portfolio system, several major structural changes have taken place at this institution. A new student information system came on line, and a new learning management system and e-mail system were implemented for students and faculty. At the same time, the university was going through a strategic planning process and defined foci around student engagement, academic services, and mission identity (including the culture of life-long learning), all of which the e-portfolio initiative supports. Naturally, these structural and infrastructural changes and reviews delayed the implementation of a new e-portfolio system. However, they also spurred the student body to take a greater interest in other potential technology changes, and the work group’s results have been reviewed by groups of adult students and traditional students alike. (The new e-portfolio system will serve both student populations, which have different curricular requirements and different practical motivations when developing an e-portfolio.¹²) The adult students have recommended that a new-generation e-portfolio be deployed with the criteria as defined, while the traditional students would like a chance to better understand the criteria and add their input to the prioritization. Their feedback will be easily incorporated due to the AHP/Decision Lens modeling process.

Conclusion

The deployment of e-portfolios is growing because of the benefits they can bring to learners, faculty, and higher-education institutions as a whole¹³:

• For students, e-portfolios help them take control of their learning by allowing them to reflect on what they have learned and plan for their careers.
• For faculty, e-portfolios allow them to view their students’ progress within the larger context of institutional requirements, students’ needs, and students’ previous achievements.
• For the institution, e-portfolios support a wider and more accurate form of assessment.

The growing interest in e-portfolios will probably continue because the next generation of e-portfolios emphasizes the "e" aspect of the name. For example, different users, such as faculty, other students, and potential employers, will have different access to and views of a student’s e-portfolio. The essence of the benefits of the next generation of e-portfolios is their potential to transform static information (remember the old binders with project reports?) into dynamic, flexible, growing information, which can be shared and viewed from different perspectives and within various contexts as needed. This next generation of e-portfolios brings together, in a single system, the functions of portfolios with reflective journals, career planning, and life-long personal development. Some e-portfolios are becoming networking tools that enhance student learning and engagement, while at the same time supporting assessment efforts based on the evaluation of student work and growth.¹⁴

The impressive list of functionalities in the new generation of e-portfolios constitutes a challenge, although by no means the only one, for institutions planning to deploy or renew their e-portfolio systems (see also "The Challenges to E-Portfolio Adoption"). To date, there is no standard definition of e-portfolio functionalities (or even a single e-portfolio definition). This makes the task of setting up relevant criteria particularly difficult. In this article the proposed criteria set for an e-portfolio focuses on both facilitating the assessment of student learning outcomes and showcasing student work. While the criteria may vary according to the institution, it is unlikely that they will vary much. For example, the criterion of aesthetics/usability (pleasant to see/easy to use) is fundamental to making sure that the e-portfolio system is used by students, faculty, and potential employers. It also reflects the importance given by this institution to showcasing students’ work for job-seeking purposes. Similarly, the second most important criterion, evaluation (also called assessment), is critical in light of the culture of assessment that drives institutions focused on improving the delivery of student learning.¹⁵

The e-portfolio framework for functional requirements can be further refined to reflect the specific needs of different institutions. For example, the tiered access criterion was subdivided into external and internal; however, each of these sub-criteria could be further subdivided if necessary. Similarly, the prioritization of functions might change as the institutional context varies (for example, an accreditation decision drives heavier assessment needs). Still, the proposed functional conceptualization and prioritization process proposed in this article allows any institution to easily adapt this framework to its own institutional goals and purposes.

In reality, the need to define and redefine both an academic institution’s purpose and corresponding functional requirements for an e-portfolio system is an ongoing process. The purpose of this article is to offer a functional criteria selection framework useful for any institution that seeks to conceptualize its functional e-portfolio requirements and plan for the next generation of e-portfolio systems. Also, the functional prioritization process described here (using the AHP/Decision Lens methodology) can be applied to the development of selection criteria for any other higher-education technological decisions,¹⁵ beyond the e-portfolio application discussed.

1. Ali Jafari, "The ‘Sticky’ ePortfolio System: Tackling Challenges and Identifying Attributes," EDUCAUSE Review, vol. 39, no. 4 (July/August 2004), pp. 38–49.
2. An excellent and succinct review of the uses and benefits of e-portfolios is given by Bob Banks in his 2004 white paper, "E-Portfolios: Their Use and Benefits," published by Tribal Technology.
3. Students were also asked to provide feedback at the time of the faculty survey in the form of focus groups; however, this method proved ineffective at the time. Subsequently, new student government leadership created greater connections between students, faculty, and the administration, and a new student e-portfolio review group was formed. The work group goal, the survey, and the results were all provided to the hosting faculty committees for review and feedback, as well as being announced during faculty assemblies to promote faculty engagement in the e-portfolio project.
4. A seminal study on this topic is "The Illusory Diffusion of Innovation: An Examination of Assimilation Gaps," by Robert G. Fichman and Chris F. Kemerer, Information Systems Research, vol. 10, no. 3 (1999), pp. 255–275.
5. Most vendors had to demonstrate their systems at least twice in addition to opening trial accounts for the group members. During the first set of demonstrations the group members learned and defined the different system characteristics. The second set of demonstrations allowed the categorization of the different characteristics into desirable functionalities. It was only during the third and last phase (including trial accounts) that prioritization was possible. The whole process took place over approximately a year.
6. Although the final results of the discussions are listed here, the process of coming up with these criteria was rather long, taking several months. The problem was that different vendors had different names for the same functions and sometimes the same names for different functions. It took several demonstrations (sometimes from the same vendor) before the e-portfolio work group was able to fully understand and categorize the functions proposed in this article.
7. Sample of current e-portfolio system’s display of a student’s artifact collection.
8. In web-related terminology, "stickiness" refers to the ability of the website or e-system (the e-portfolio in this case) to retain the interest of the users, to the point that they return to using the site or system over and over.
9. The Technology Adoption Model (TAM) proposed by Fred D. Davis in "Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology," MIS Quarterly, vol. 13, no. 3 (September 1989), pp. 319–340, states that two important antecedents leading to the intention to use a new technology are the perceived usefulness and the perceived ease of use of the new technology.
10. For a detailed explanation of this prioritization process, refer to Thomas L. Saaty, The Analytic Hierarchy Process (Pittsburgh: RWS Publications, 1982).
11. The weights were calculated using the Decision Lens software and the Analytic Hierarchy Process approach. In this method, the decision makers compare the criteria pairwise and express their relative preferences in terms of numerical values using a 1-to-9 scale (for example, decision makers could state that Usability is strongly more important than Areas for Student Reflection with an intensity of 5). The final priorities (importance weights) are calculated based on these numerical preferences.
12. Adults understand the e-portfolio from a professional/career-based standpoint. Traditional age students may desire to do this more from a fun "ego" portfolio standpoint — that is, more show, less substance. "Ego" portfolio was a term used in a recent Electronic Portfolio Action and Communication (EPAC) webcast — not as a pejorative but as a hook.
13. Elizabeth Clark and Bret Eynon, "E-portfolios at 2.0 — Surveying the Field," Peer Review, vol. 11, no. 1 (Winter 2009), pp.18–23.
14. We do not think, as some people have argued, that e-portfolios will face a challenge from other social network tools such as Facebook or MySpace due to the specific purpose and specialized nature of e-portfolios. It is for this same reason that multiple different social networking systems coexist. They are all useful for different purposes.
15. Other institutions may need e-portfolios primarily for accreditation. Not in this case. The goal is to improve student learning first. It happens that it also supports the institution’s accreditation efforts.
16. See for example Thomas L. Saaty and Luis Gonzalez Vargas, Decision Making in Economic, Social, Political, and Technological Environments with the Analytic Hierarchy Process (Pittsburgh: RWS Publications, 1994).

© 2010 Enrique Mu, Sallie Wormer, Roberta Foizey, Beverly Barkon, and Mark Vehec. The text of this article is licensed under the Creative Commons Attribution 3.0 license.