INTRODUCTION

This article describes a participative, action evaluation process (Nouwens & Robinson, 1991; Wadsworth, 1991) that was developed when Central Queensland University (CQU) in Australia was forced to replace existing university-wide Learning Management System (LMS) software used to support the development and delivery of online courses. A number of important lessons were learned during this process and this article offers a sociotechnical systems analysis (Owens, 1998) of this evaluation process to assist academic staff involved in evaluation and choice of similar enterprise systems to understand the complexities and implications of decisions they make.

HISTORY

In 1995, the CQU had 8357 effective full-time students, about half the students studying in the distance education (Central Queensland University, 2001). Previously there had been some computer based support for distance learners, particularly using email. From 1995 to 1999, a small but increasing number of online courses began to be developed to supplement distance and face-to-face modes of study. Long experience with print-based distance education dating back to 1978, and the early, gradual introduction of online courses, supported a steady development of online infrastructure and a progressive development of the technical and online teaching skills.

Growing interest in online teaching as a way to expand postgraduate programs into niche markets and to provide distance and on-campus students with additional resources and support made it difficult for the University’s multimedia development centre to support the increasing demand for online courses. In 1999, the University began to investigate the adoption of a Learning Management System to enable teaching staff to develop and manage online courses with little professional support. The term Learning Management System (LMS) refers to an integrated set of networked, computerised tools that support online learning (Virtual Learning Environment or Course Management System are other terms that are sometimes used). Typically, these systems include tools like discussion lists and chat rooms for online communication, automated quizzes and other assessment tools, tools that make it easier to prepare content for online presentation and some tools to monitor and manage student activities.

A survey and a simple, technical evaluation of learning management software available in 1999 led to the choice of WebCT for major LMS trials. It had the functionality required, the cost was low, it was well supported, it was compatible with the University’s IT infrastructure and it promised to allow more efficient production. Centralised funding and support led to a rapid up take and, by late 2002, 231 courses had been developed to provide online course resources and/or support.

Then, early in 2002, the University was informed that its then current version of WebCT would not be supported after 2003. Either of the alternative versions of WebCT would involve some increase in cost and an allocation of resources to support the change. CQU was in a position where external forces (WebCT software changes) required a major, unplanned change and a major reallocation of resources to support the change. In contrast to the brief evaluation in 1999, in 2002 there was a much wider selection of sophisticated LMS products on the market and CQU had made a significant investment in the use of an LMS, both in monetary terms and in the time and commitment of academic and support staff in preparing online resources. Three years of experience had exposed shortcomings of the system being used, as well as the ‘risk’ attached to a commitment to the use of an ‘off the shelf’ LMS. However, there was still a growing demand for the benefits offered by an LMS in the development of online courses.

This evaluation did not seek to make judgements about the effectiveness of an LMS as a means for improving the effectiveness of learning in higher education; it followed the approach of Dutton, Cheong and Park (2004), whose case study inquires into the “variety of technical, institutional, social and economic constraints on the innovation process” (p. 69). However, this (CQU) study focuses not on the initial introduction of an institution-wide LMS and the major changes that entails, but on the question of whether and how to change from one major enterprise LMS to another.

THE EVALUATION APPROACH AND ITS INTERPRETATION

The authors participated in the LMS evaluation process as project manager and facilitator, and were responsible for formulating and planning the evaluation approach and for managing the evaluation project to the point of choosing the LMS. Traditionally evaluation in education has had a summative focus, seeking to assemble information on which to make a judgement about the worth of some activity, tool or program (Smith & Lovat, 2004; Worthen, Sanders & Fitzpatrick, 1997). Such judgements have traditionally been made by experts, people from outside the organisation who can look objectively at the situation. Such experts obtain legitimacy from the application of analytical-empirical methodologies; they have what Habermas (1996) calls a “technical interest” in the evaluation. Smith and Lovat (2004, p. 188) argue that for such expert evaluations “the meanings and interpretations constructed by the evaluators were not accurate representations of the perceptions of the participants....There is no one reality of a situation”. Thus the model for valid and reliable evaluation in educational contexts has moved from a purely technical interest in the use of independent, external experts who objectively gather, analyse and make recommendations towards a practical interest in which participants in the context being evaluated negotiate and interpret meaning, and are supported to participate in and control the evaluation process and its outcomes.

The transition from an expert evaluation to a participative evaluation model is consistent with an understanding that the culture of an organisation is not coherent and unitary, and that large organisations like universities actually contain within them many sub-cultures that both support and conflict with one another (Luck, Jones, McConachie & Danaher, 2004; Schein, 1996). In this instance, the use of a participative evaluation process gave a voice to the main groups (sub-cultures) which had a strong interest in the evaluation process. Four main groups were identified: academics, information technology staff, multimedia development specialists and managers/administrators. Owing to limited time and the intervention of the summer vacation period, students were not included in the evaluation but their interests were represented by the academic evaluation group.

To illuminate the evaluation approach described above, this article uses a sociotechnical systems framework to inform the interpretation of this process. Sociotechnical systems theory provides a framework for interpreting and understanding the interactions that occur in organisations subject to technological change. Owens (1998) provides a useful model that invites an exploration of interactions between four major interacting activity subsystems within an educational organisation (humans, structures, tasks, technologies) on the one hand, and the goals of the organisation and the contingencies of the environment in which the organisation operates on the other. (See Figure 1.)

Figure 1: The educational institution as a sociotechnical system
(Adapted from Owens, 1998, p. 78)

Owens’ model is supported by more recent studies of the use of LMSs and the concept of the social shaping of technology (Dutton et al., 2004) that highlight organisational, cultural, economic and other factors influencing the process of technological change. The six components of Owens’ model can be used to interrogate the situations facing the four interest groups (academic, technical, development support and administration) that were given a voice in the LMS evaluation process, and the University as a whole.

Owens presents the four organisational subsystems, plus organisational goals and environmental contingencies, as components that exist in a dynamic, organic equilibrium. If changes occur or are intended in one component of the model, we should be mindful of responses that change generates in other components. In other words, if a decision is made to change the technology the University uses (i.e. WebCT), one should consider the consequences for changes in the tasks performed in the organisation, the effect on people and their skills and interests, the organisational structures in which people operate and the effects on other technologies already in use. The impact of the change on the goals of the University (e.g. resource allocation or marketing) and the response of the external environment to such a change should also be considered (e.g. the ability to attract additional students or additional funds or enrolments from industry).

METHOD

As described earlier, the University faced a version-change WebCT in the short-term. This would have involved some installation work and systems conversion work for technicians but little work in conversion of content. However, WebCT indicated that the life of the new version could be limited and that an upgrade to a more expensive enterprise system (Vista) would be required. These prospective changes, the trial nature of the original commitment to WebCT, pressure from some Faculties to move to other LMSs, problems experienced with WebCT in the CQU context and the fact that staff had developed some experience with the operation of a LMS led to a feeling among current LMS users that the University should conduct a rigorous evaluation of LMSs including an assessment of the issues and costs involved in changing to another LMS.

Timeframe
During the initial stages of the evaluation, project managers identified a shortlist of four LMSs for detailed evaluation based on lists of essential criteria developed by the four interest groups. Initial work with representatives from these groups established project milestones that would provide technical staff with time to install and test the new system, provide time for a trial to develop and deliver a small number of courses to students and provide time to convert all existing online courses to the new LMS for delivery in 2004. Only two months could be allowed for the evaluation of shortlisted LMSs, with another month set aside for reporting and the final decision to be reached.

Stakeholders in the Change Process
While senior management of the University was informed early in 2002 that a decision had to be made about choosing an LMS for 2004, no priority for action was given until the issue was raised at a staff development forum in November of that year. Teaching staff indicated that they required at least 12 months notice of such a change to allow them to develop appropriate skills and resources for new courses. This illustrates the different perspectives and priorities that can arise between what Luck et al. (2004) call the management sub-culture and the academic sub-culture.

Pressure for change came from the grass-roots users of the LMS. To harness the energy for change and to build commitment to the final decision, project managers decided to involve critical stakeholder user-groups in a participatory action evaluation process to conduct a summative evaluation of commercially available LMSs and to make a recommendation to senior management to guide the decision to choose the University LMS for 2004 and some time into the future. The project manager’s task was to involve representatives of the four critical stakeholder groups (academic, technical, development support and management) in the evaluation and decision-making process at three levels:

1. In the plenary group with representatives of all four groups.
2. In each of the four groups as they conducted separate evaluations of their facet of the four shortlisted LMSs.
3. In the final collaborative process in which the four groups would justify and develop a recommendation with active involvement of senior management who carried final responsibility for the selection of the LMS.

To make best use of the available time, each interest group worked largely independently to develop its own evaluation criteria appropriate for its area of expertise, and implement suitable testing and evaluation processes for the four shortlisted LMSs.

A team of computer administrators looked at technical issues such as the reliability and robustness of the software, technical support and the LMSs’ compatibility with other CQU computer systems and industry technical standards. CQU’s multimedia development unit investigated the potential of each LMS for specialised development, IMS standards and implications for conversion of a large number of courses currently existing in WebCT. The administration group investigated contract conditions, pricing, commercial viability of the vendor, planned version changes and related commercial matters. The fourth group consisted of teaching academics and focussed on a range of issues that related to teaching and learning. The timing of the evaluation made it unfeasible to include students as members of this working party, but the teaching staff undertook to represent this group.

While the management, technical and development stakeholder sub-cultures consisted of relatively small numbers of people, the academic issues group, representing the interests of a large group of disparate users, had first to consult academic staff to determine what issues were seen as important and to promote ownership of the outcome.

The effort required by teaching staff to learn to use a different LMS application was one concern which has since been identified as a factor in reducing academics’ motivation to teach online (Dutton et al., 2004). The academic working party evaluation team identified this as a major issue that was particularly of concern to those who had spent considerable time developing skills in WebCT. A number of institutional changes that had occurred at CQU in the preceding years had left the staff change-weary (Luck et al., 2004) and there were concerns about whether staff would support yet another change. Thus ease of use was one of the key criteria that emerged from this group.

Decision-Making Process
Evaluations involved a range of methods including where appropriate tests and trials of the LMSs, surveys, consultations with other users and vendor presentations. Each group prepared a written report indicating the evaluation process and criteria used, the justification of the criteria, the ranking of each LMS and the justification of the ranking. All parties then met with senior management to discuss the issues and to debate the advantages and disadvantages of various options, including the option to continue with the new version of WebCT. The final decision, to change to a different LMS, Blackboard, was made late in March 2003.

Since the decision, installation and trials have proceeded successfully, with Blackboard in full operation for the March-June 2004 teaching period. Formative evaluation processes, including research and user forums, development teams and staff development for academic staff, are supporting current and future developments.

DISCUSSION OF REFLECTIONS ON THE PROJECT

Critical reflection on practice is an important element in the action evaluation process (Wadsworth, 1991). Organisational lessons learned by the evaluation team from this participatory evaluation process are outlined in the following paragraphs.

This project demonstrated how vulnerable universities are likely to become to external decisions made by an LMS vendor to change or withdraw its product This evaluation process involved a holistic evaluation of both LMS software and the costs and benefits of changing the LMS at CQU. In just three years, the University had developed a substantial investment with over 230 courses using WebCT; however online access remains optional for most of these courses which are offered face-to-face and in print-based distance mode. As institutions develop a greater dependence on online learning, they should assess the risks of becoming ‘locked-in’ to particular LMS software when the cost of change becomes too great. Owens’ (1998) framework invites one to question how responsive a sociotechnical system can be to changes in the external environment if it becomes tied to LMS software and a vendor who does not develop the software in the direction the institution desires to go. Discussions and reflection on this issue in plenary sessions explored the notion of ‘middle-ware’, a university controlled interface between students and the learning management software that could continue to give an institution a consistent interface in spite of changes from one LMS to another.

Difficulty in transferring content from one LMS to another owing to conflicting data types is another concern that can be addressed by linking the content through a separate mediating system (Chapman, 2003). However, this approach of layering technological systems is likely to increase the impact on the humans, the structures and tasks and the goals of a university. Accordingly, one would expect that the sub-cultures that exist in a traditional university (Luck et al., 2004) would resist the changes that would make such technological change effective. The challenge for change agents is how to engage these sub-cultures in the processes of change while protecting the core values of education.

It was useful to recognise a university as a complex organisation comprised of different sub-cultures that can operate semi-autonomously in a static world, but that need active intervention to help them work together in a dynamic and changing environment
This difference in perspectives between senior management and academic course developers about timelines for decision-making regarding the choice of an LMS illustrates the point made by Luck et al. (2004) that one’s understanding of an organisation can be enhanced by considering it not as a coherent culture, but as a collection of groups with different sub-cultures in collaboration and in conflict. A comment by one of the technical group suggesting that we should seek to change people’ behaviour because information technology systems are difficult to change reinforces this point.

Because the pressure for change came from grass-roots users and those with interests in the future use of an LMS, it was recognised that the legitimacy of the evaluation and the acceptance of the final decision depended on open and transparent processes that involved all critical stakeholder groups. The critical stakeholders are those who control parts of the process of using the LMS. Cultural Historical Activity Theory (CHAT) (Hasan, Gould, Larkin & Vrazalic, 2001) suggests that organisations can be understood as communities of human activity that function effectively by the division of labour guided by formal and informal organisational rules.

This notion of the division of labour within an organisation is echoed in Owens’ (1998) role of the task in sociotechnical systems, but focuses attention on the interdependence of tasks, and the influence that organisational sub-cultures (Luck et al., 2004) have on the flow of organisational activity. In a simple traditional university and a static operating environment, there is time to deal with problems, change practices and manage from the top of the organisation to overcome the reluctance of sub-cultures to change. A complex organisation like CQU, with 13 campuses, multiple modes of study and a diverse range of domestic and international students, and under government pressure to change, requires a university where the critical groups have sub-cultures of collaboration in order to obtain the benefits of change rather than cultures of resistance. The participative, action-evaluation approach adopted in this project can be used to support such organisational development.

Evaluation in Dynamic Settings Should Involve Key Participant Groups
Batchelor and Maxwell (1987) propose “a process in which the ‘practitioners’ are included as evaluators, which features collaborative planning and data-gathering, self-reflections and responsiveness” (p. 70). This principle of stakeholder involvement was a high priority for the evaluation process, not only to ensure that all relevant issues were identified and taken into account, but also to pave the way for the successful implementation of the chosen system. Acceptance of and even commitment to change is enhanced when “‘Ownership’ of the evaluation is vested in the ‘Practitioners’” (Batchelor & Maxwell, 1987, p. 70).

Involvement of the critical interest groups in the evaluation and the decision-making process led to general acceptance of the decision to change and to undertake the work associated with the change. The process also helped the major stakeholder groups (academic, technical, development, administration) to understand better the perspectives of each of these sub-cultures within the University and the labour contributed by each ‘division’ of the educational enterprise. There are plans to carry this participant action-evaluation process into formative evaluation of the development of online courses, to promote scholarly, evidence-based practice models that involve all stakeholders in conversation and common collaborative effort.

Successful change is promoted by a three-way balance among sub-cultural group interests: an interest group to drive change, bottom-up participant action and top-down support for change
This project was initiated by a particular interest group (academic) that drives the University to respond to a problem. This group then proceeded to enlist key members from other critical groups in a bottom-up process to achieve change. Implications of the problem were articulated to obtain the support of senior management, which accepted the proposed evaluation plan.

Other projects that have not proceeded so well have lacked these three components: a driving group, the participation of key interest groups and senior management support. Projects driven from the top or driven by an interest group without participant involvement seem to have a long and difficult gestation and short life. Warne, Ali, Linger and Pascoe (2003, p. 307) make this point by drawing attention to the organisational conflict of understanding that arises between the practitioner-team’s experiential understanding of a task and the abstract understanding of senior management. Smith and Lovat (2004) maintain that participation evaluation is effective because only the participants in a context really understand the detail and because ‘experts’ more distant from an activity often fail to interpret the cultural complexity of organisational operations. This LMS evaluation project shows that a participative decision-making process can be effective and can be completed quickly.

Academic staff tend to revert to the pedagogy they normally
use in face-to-face or distance education
While the LMS evaluation, the LMS choice and the conversion process have been largely successful; encouraging academics to make both efficient and pedagogically effective use of the web for teaching has proved a greater challenge. The University seems to understand the need to direct resources to the evaluation and conversion process and to the purchase of software, but seems to assume that academic staff will use the system well. Dutton et al. (2004) have found that, when teaching online, many academics will default to traditional patterns of teaching. Owens’ (1998) framework makes it clear that changing technology alone will not produce effective change; he argues that the tasks academics do, the structures they operate under and the way they relate to other people in the system must all change in a coherent way to make effective use of the new technology. The idea that sub-cultures within the University are likely to resist change (Luck et al., 2004) adds to the problem of change, and suggests that a participative search for solutions would be more likely to be effective.

CONCLUSION

The CQU experience shows that changing an LMS has major consequences for academic, administration, technical and multimedia development staff. Evaluation to choose a new LMS should draw on the experience of all these groups and provides an opportunity for these groups to understand better the perspectives of the other groups. Substantial university resources are absorbed by such a change, yet the need to change may be beyond the University’s control. Institutions should monitor the health and intentions of the vendor of their LMS software to obtain warning of change and manage risks associated with a sudden change of LMS.

Owens’ (1998) sociotechnical systems model highlights the need to involve the critical organisational units in evaluation and decision-making. It also draws attention to the impact that technology changes will continue to have on teaching tasks, university structures, participants and the goals of the university as LMS software and the web continue to develop. In this dynamic technological environment, the choice of technology is never a ‘final’ decision leading to a separate implementation phase. New problems can be expected to arise as the different interest groups learn to use new and more complex LMSs and identify strengths and weaknesses that were not apparent in the initial evaluation or have been introduced in upgrades. At CQU, regular user forums involving all the participant groups are conducted in a participative, formative evaluation process to address ongoing software and organisational change and to improve the LMS.

REFERENCES

Batchelor, M., & Maxwell, T. (1987). Action evaluation. New Education: International Journal of Educational Theory and Practice, 9(1-2), 70-77.

Chapman, B. (2003). LCMS report: Comparing analysis of enterprise Learning Content Management Systems. Sunnyvale CA: Brandon-Hall.

Central Queensland University. (2001). Central Queensland University student profile 1995-2000. Rockhampton: Analysis and Planning Unit, Central Queensland University.

Dutton, W. H., Cheong, P. H., & Park, N. (2004). The social shaping of a Virtual Learning Environment: The case of a university-wide course management system. Electronic Journal of e-Learning, 2(1), 69-80.

Habermas, J. (1996). Knowledge and human interests: A general perspective. In W. Outhwaite (Ed.), The Habermas reader (pp. 96-204). Cambridge, UK: Polity Press.

Hasan, H., Gould, E., Larkin, P., & Vrazalic, L. (Eds.). (2001). Information systems and Activity Theory. Wollongong, NSW: University of Wollongong Press.

Luck, J., Jones, D., McConachie, J., & Danaher, P. A. (2004, February 18). Challenging enterprises and subcultures: Interrogating ‘best practice’ in Central Queensland University’s course management systems. Paper presented at the Teaching and Learning Showcase, Central Queensland University, Rockhampton, Qld.

Nouwens, F., & Robinson, P. (1991). Evaluation and the development of quality learning materials. Australian Journal of Educational Technology, 7(2), 93-116.

Owens, R. G. (1998). Organisational behavior in education (6th ed.). Boston, MA: Allyn & Bacon.

Schein, E. H. (1996). Three cultures in management: The key to organisational learning. Sloan-Management Review, 38(1), 9-20.

Smith, D. L., & Lovat, T. J. (2004) Curriculum: Action on reflection (4th ed.). Tuggeragh, NSW: Social Science Press.

Wadsworth, Y. (1991). Everyday evaluation on the run. Melbourne, Vic: Action Research Issues.

Warne, L., Ali, I., Linger, H., & Pascoe, C. (2003). Sociotechnical foundations for knowledge management. In H. Hasan & M. Handzic (Eds.), Australian studies in knowledge management (pp. 277-321). Wollongong, NSW: University of Wollongong Press.

Worthen, B., Sanders, J., & Fitzpatrick, J. (1997). Program evaluation: Alternative approaches and practical guidelines (2nd ed.). New York: Longman.

Contact addresses of authors:

Phillipa Sturgess
Teaching and Learning Advisor
Central Queensland University, Yaamba Road, North Rockhampton, Queensland, Australia, 4701
E-mail: p.sturgess@cqu.edu.au
Phone: +61 7 4930 9116 and Fax: +61 7 4930 9792

Fons Nouwens
Position: Lecturer, Learning Evaluation Innovation and Development Centre
Institution: Central Queensland University, Bruce Highway, North Rockhampton, Queensland, Australia, 4702
E-mail: f.nouwens@cqu.edu.au
Personal web link: http://dtls.cqu.edu.au/leid_centre/team.html
Phone (office): +61 7 4930 9577
Fax: +61 7 4930 9792

Interests of authors:
Phillipa Sturgess interests with encouraging and supporting reflective teaching practice, particularly as it pertains to online learning and the development of effective online learning environments; interaction between online systems and the teachers and learners using them; informal learning in community-based contexts.

Fons Nouwens interests with curriculum development, generic skills, graduate attributes, expansive learning, open and distance learning, evaluation, instructional design, writing process, online learning and teaching, lifelong learning.