Turkish Online Journal of Distance Education-TOJDE Julyl 2004 ISSN 1302-6488 Volume: 5 Number: 3
Evaluation of Online Learning Management Systems
This article describes a participative, action evaluation process (Nouwens & Robinson, 1991; Wadsworth, 1991) that was developed when Central Queensland University (CQU) in Australia was forced to replace existing university-wide Learning Management System (LMS) software used to support the development and delivery of online courses. A number of important lessons were learned during this process and this article offers a sociotechnical systems analysis (Owens, 1998) of this evaluation process to assist academic staff involved in evaluation and choice of similar enterprise systems to understand the complexities and implications of decisions they make.
In 1995, the CQU had 8357 effective full-time students, about half the students studying in the distance education (Central Queensland University, 2001). Previously there had been some computer based support for distance learners, particularly using email. From 1995 to 1999, a small but increasing number of online courses began to be developed to supplement distance and face-to-face modes of study. Long experience with print-based distance education dating back to 1978, and the early, gradual introduction of online courses, supported a steady development of online infrastructure and a progressive development of the technical and online teaching skills.
Growing interest in online teaching as a way to expand postgraduate programs into niche markets and to provide distance and on-campus students with additional resources and support made it difficult for the University’s multimedia development centre to support the increasing demand for online courses. In 1999, the University began to investigate the adoption of a Learning Management System to enable teaching staff to develop and manage online courses with little professional support. The term Learning Management System (LMS) refers to an integrated set of networked, computerised tools that support online learning (Virtual Learning Environment or Course Management System are other terms that are sometimes used). Typically, these systems include tools like discussion lists and chat rooms for online communication, automated quizzes and other assessment tools, tools that make it easier to prepare content for online presentation and some tools to monitor and manage student activities.
A survey and a simple, technical evaluation of learning management software available in 1999 led to the choice of WebCT for major LMS trials. It had the functionality required, the cost was low, it was well supported, it was compatible with the University’s IT infrastructure and it promised to allow more efficient production. Centralised funding and support led to a rapid up take and, by late 2002, 231 courses had been developed to provide online course resources and/or support.
Then, early in 2002, the University was informed that its then current version of WebCT would not be supported after 2003. Either of the alternative versions of WebCT would involve some increase in cost and an allocation of resources to support the change. CQU was in a position where external forces (WebCT software changes) required a major, unplanned change and a major reallocation of resources to support the change. In contrast to the brief evaluation in 1999, in 2002 there was a much wider selection of sophisticated LMS products on the market and CQU had made a significant investment in the use of an LMS, both in monetary terms and in the time and commitment of academic and support staff in preparing online resources. Three years of experience had exposed shortcomings of the system being used, as well as the ‘risk’ attached to a commitment to the use of an ‘off the shelf’ LMS. However, there was still a growing demand for the benefits offered by an LMS in the development of online courses.
This evaluation did not seek to make judgements about the effectiveness of an LMS as a means for improving the effectiveness of learning in higher education; it followed the approach of Dutton, Cheong and Park (2004), whose case study inquires into the “variety of technical, institutional, social and economic constraints on the innovation process” (p. 69). However, this (CQU) study focuses not on the initial introduction of an institution-wide LMS and the major changes that entails, but on the question of whether and how to change from one major enterprise LMS to another.
THE EVALUATION APPROACH AND ITS INTERPRETATION
The authors participated in the LMS evaluation process as project manager and facilitator, and were responsible for formulating and planning the evaluation approach and for managing the evaluation project to the point of choosing the LMS. Traditionally evaluation in education has had a summative focus, seeking to assemble information on which to make a judgement about the worth of some activity, tool or program (Smith & Lovat, 2004; Worthen, Sanders & Fitzpatrick, 1997). Such judgements have traditionally been made by experts, people from outside the organisation who can look objectively at the situation. Such experts obtain legitimacy from the application of analytical-empirical methodologies; they have what Habermas (1996) calls a “technical interest” in the evaluation. Smith and Lovat (2004, p. 188) argue that for such expert evaluations “the meanings and interpretations constructed by the evaluators were not accurate representations of the perceptions of the participants....There is no one reality of a situation”. Thus the model for valid and reliable evaluation in educational contexts has moved from a purely technical interest in the use of independent, external experts who objectively gather, analyse and make recommendations towards a practical interest in which participants in the context being evaluated negotiate and interpret meaning, and are supported to participate in and control the evaluation process and its outcomes.
The transition from an expert evaluation to a participative evaluation model is consistent with an understanding that the culture of an organisation is not coherent and unitary, and that large organisations like universities actually contain within them many sub-cultures that both support and conflict with one another (Luck, Jones, McConachie & Danaher, 2004; Schein, 1996). In this instance, the use of a participative evaluation process gave a voice to the main groups (sub-cultures) which had a strong interest in the evaluation process. Four main groups were identified: academics, information technology staff, multimedia development specialists and managers/administrators. Owing to limited time and the intervention of the summer vacation period, students were not included in the evaluation but their interests were represented by the academic evaluation group.
To illuminate the evaluation approach described above, this article uses a sociotechnical systems framework to inform the interpretation of this process. Sociotechnical systems theory provides a framework for interpreting and understanding the interactions that occur in organisations subject to technological change. Owens (1998) provides a useful model that invites an exploration of interactions between four major interacting activity subsystems within an educational organisation (humans, structures, tasks, technologies) on the one hand, and the goals of the organisation and the contingencies of the environment in which the organisation operates on the other. (See Figure 1.)
Figure 1: The educational institution as a sociotechnical
Owens presents the four organisational subsystems, plus organisational goals and environmental contingencies, as components that exist in a dynamic, organic equilibrium. If changes occur or are intended in one component of the model, we should be mindful of responses that change generates in other components. In other words, if a decision is made to change the technology the University uses (i.e. WebCT), one should consider the consequences for changes in the tasks performed in the organisation, the effect on people and their skills and interests, the organisational structures in which people operate and the effects on other technologies already in use. The impact of the change on the goals of the University (e.g. resource allocation or marketing) and the response of the external environment to such a change should also be considered (e.g. the ability to attract additional students or additional funds or enrolments from industry).
As described earlier, the University faced a version-change WebCT in the short-term. This would have involved some installation work and systems conversion work for technicians but little work in conversion of content. However, WebCT indicated that the life of the new version could be limited and that an upgrade to a more expensive enterprise system (Vista) would be required. These prospective changes, the trial nature of the original commitment to WebCT, pressure from some Faculties to move to other LMSs, problems experienced with WebCT in the CQU context and the fact that staff had developed some experience with the operation of a LMS led to a feeling among current LMS users that the University should conduct a rigorous evaluation of LMSs including an assessment of the issues and costs involved in changing to another LMS.
Stakeholders in the Change Process
Pressure for change came from the grass-roots users of the LMS. To harness the energy for change and to build commitment to the final decision, project managers decided to involve critical stakeholder user-groups in a participatory action evaluation process to conduct a summative evaluation of commercially available LMSs and to make a recommendation to senior management to guide the decision to choose the University LMS for 2004 and some time into the future. The project manager’s task was to involve representatives of the four critical stakeholder groups (academic, technical, development support and management) in the evaluation and decision-making process at three levels:
To make best use of the available time, each interest group worked largely independently to develop its own evaluation criteria appropriate for its area of expertise, and implement suitable testing and evaluation processes for the four shortlisted LMSs.
A team of computer administrators looked at technical issues such as the reliability and robustness of the software, technical support and the LMSs’ compatibility with other CQU computer systems and industry technical standards. CQU’s multimedia development unit investigated the potential of each LMS for specialised development, IMS standards and implications for conversion of a large number of courses currently existing in WebCT. The administration group investigated contract conditions, pricing, commercial viability of the vendor, planned version changes and related commercial matters. The fourth group consisted of teaching academics and focussed on a range of issues that related to teaching and learning. The timing of the evaluation made it unfeasible to include students as members of this working party, but the teaching staff undertook to represent this group.
While the management, technical and development stakeholder sub-cultures consisted of relatively small numbers of people, the academic issues group, representing the interests of a large group of disparate users, had first to consult academic staff to determine what issues were seen as important and to promote ownership of the outcome.
The effort required by teaching staff to learn to use a different LMS application was one concern which has since been identified as a factor in reducing academics’ motivation to teach online (Dutton et al., 2004). The academic working party evaluation team identified this as a major issue that was particularly of concern to those who had spent considerable time developing skills in WebCT. A number of institutional changes that had occurred at CQU in the preceding years had left the staff change-weary (Luck et al., 2004) and there were concerns about whether staff would support yet another change. Thus ease of use was one of the key criteria that emerged from this group.
Since the decision, installation and trials have proceeded successfully, with Blackboard in full operation for the March-June 2004 teaching period. Formative evaluation processes, including research and user forums, development teams and staff development for academic staff, are supporting current and future developments.
DISCUSSION OF REFLECTIONS ON THE PROJECT
Critical reflection on practice is an important element in the action evaluation process (Wadsworth, 1991). Organisational lessons learned by the evaluation team from this participatory evaluation process are outlined in the following paragraphs.
This project demonstrated how vulnerable universities are likely to become to external decisions made by an LMS vendor to change or withdraw its product This evaluation process involved a holistic evaluation of both LMS software and the costs and benefits of changing the LMS at CQU. In just three years, the University had developed a substantial investment with over 230 courses using WebCT; however online access remains optional for most of these courses which are offered face-to-face and in print-based distance mode. As institutions develop a greater dependence on online learning, they should assess the risks of becoming ‘locked-in’ to particular LMS software when the cost of change becomes too great. Owens’ (1998) framework invites one to question how responsive a sociotechnical system can be to changes in the external environment if it becomes tied to LMS software and a vendor who does not develop the software in the direction the institution desires to go. Discussions and reflection on this issue in plenary sessions explored the notion of ‘middle-ware’, a university controlled interface between students and the learning management software that could continue to give an institution a consistent interface in spite of changes from one LMS to another.
Difficulty in transferring content from one LMS to another owing to conflicting data types is another concern that can be addressed by linking the content through a separate mediating system (Chapman, 2003). However, this approach of layering technological systems is likely to increase the impact on the humans, the structures and tasks and the goals of a university. Accordingly, one would expect that the sub-cultures that exist in a traditional university (Luck et al., 2004) would resist the changes that would make such technological change effective. The challenge for change agents is how to engage these sub-cultures in the processes of change while protecting the core values of education.
It was useful to recognise a university
as a complex organisation comprised of different sub-cultures that can
operate semi-autonomously in a static world, but that need active intervention
to help them work together in a dynamic and changing environment
Because the pressure for change came from grass-roots users and those with interests in the future use of an LMS, it was recognised that the legitimacy of the evaluation and the acceptance of the final decision depended on open and transparent processes that involved all critical stakeholder groups. The critical stakeholders are those who control parts of the process of using the LMS. Cultural Historical Activity Theory (CHAT) (Hasan, Gould, Larkin & Vrazalic, 2001) suggests that organisations can be understood as communities of human activity that function effectively by the division of labour guided by formal and informal organisational rules.
This notion of the division of labour within an organisation is echoed in Owens’ (1998) role of the task in sociotechnical systems, but focuses attention on the interdependence of tasks, and the influence that organisational sub-cultures (Luck et al., 2004) have on the flow of organisational activity. In a simple traditional university and a static operating environment, there is time to deal with problems, change practices and manage from the top of the organisation to overcome the reluctance of sub-cultures to change. A complex organisation like CQU, with 13 campuses, multiple modes of study and a diverse range of domestic and international students, and under government pressure to change, requires a university where the critical groups have sub-cultures of collaboration in order to obtain the benefits of change rather than cultures of resistance. The participative, action-evaluation approach adopted in this project can be used to support such organisational development.
Evaluation in Dynamic Settings
Should Involve Key Participant Groups
Involvement of the critical interest groups in the evaluation and the decision-making process led to general acceptance of the decision to change and to undertake the work associated with the change. The process also helped the major stakeholder groups (academic, technical, development, administration) to understand better the perspectives of each of these sub-cultures within the University and the labour contributed by each ‘division’ of the educational enterprise. There are plans to carry this participant action-evaluation process into formative evaluation of the development of online courses, to promote scholarly, evidence-based practice models that involve all stakeholders in conversation and common collaborative effort.
Successful change is promoted by
a three-way balance among sub-cultural group interests: an interest group
to drive change, bottom-up participant action and top-down support for
Other projects that have not proceeded so well have lacked these three components: a driving group, the participation of key interest groups and senior management support. Projects driven from the top or driven by an interest group without participant involvement seem to have a long and difficult gestation and short life. Warne, Ali, Linger and Pascoe (2003, p. 307) make this point by drawing attention to the organisational conflict of understanding that arises between the practitioner-team’s experiential understanding of a task and the abstract understanding of senior management. Smith and Lovat (2004) maintain that participation evaluation is effective because only the participants in a context really understand the detail and because ‘experts’ more distant from an activity often fail to interpret the cultural complexity of organisational operations. This LMS evaluation project shows that a participative decision-making process can be effective and can be completed quickly.
Academic staff tend to revert to
the pedagogy they normally
The CQU experience shows that changing an LMS has major consequences for academic, administration, technical and multimedia development staff. Evaluation to choose a new LMS should draw on the experience of all these groups and provides an opportunity for these groups to understand better the perspectives of the other groups. Substantial university resources are absorbed by such a change, yet the need to change may be beyond the University’s control. Institutions should monitor the health and intentions of the vendor of their LMS software to obtain warning of change and manage risks associated with a sudden change of LMS.
Owens’ (1998) sociotechnical systems model highlights the need to involve the critical organisational units in evaluation and decision-making. It also draws attention to the impact that technology changes will continue to have on teaching tasks, university structures, participants and the goals of the university as LMS software and the web continue to develop. In this dynamic technological environment, the choice of technology is never a ‘final’ decision leading to a separate implementation phase. New problems can be expected to arise as the different interest groups learn to use new and more complex LMSs and identify strengths and weaknesses that were not apparent in the initial evaluation or have been introduced in upgrades. At CQU, regular user forums involving all the participant groups are conducted in a participative, formative evaluation process to address ongoing software and organisational change and to improve the LMS.
Batchelor, M., & Maxwell, T. (1987). Action evaluation.
New Education: International Journal of Educational Theory and Practice,
Contact addresses of authors:
Interests of authors:
Fons Nouwens interests with curriculum development, generic skills, graduate attributes, expansive learning, open and distance learning, evaluation, instructional design, writing process, online learning and teaching, lifelong learning.