Introduction

The first decade of the 21st century is swiftly experiencing an increased need for learning that has greatly exceeded present delivery capacity of traditional educational settings. To help meet this need, many educational institutions are now offering courses as well as entire degree programs through distance education. Thus, institutions are thrust into being confronted with a rapidly changing social as well as economic scenario; and if they are to survive, they must evolve. Crucial to this successful evolution is the formulation of new assessment strategies for distance education (Stallings, 2002). Because of the process and practice of assessment which increasingly has become the focus of the public, politicians, and legislators, future historians may view the 1980s as the decade in which problems in education achieved national significance, and the 1990s as the decade of accountability (Seward, in Stallings, 2002).

The federal government, accrediting agencies, new technologies, as well as complaints from colleges about the accreditation process are all forces driving the push to reevaluate how we assess what students learn. The problem is how do we meet such broad mandates? Because distance education is relatively new, critics hold it to a higher standard as opposed to traditional education. Thus, since distance education has more to prove, educators are trying to establish a variety of assessments that prove its effectiveness (Ewell, in Carnevale, 2001). Accrediting agencies are encouraging colleges to move beyond the traditional measures of success such as satisfaction surveys from students, employers, and alumni. They stress the use of a variety of measurement tools to audit students’ work such as examining their writing and critical-thinking skills. However, many faculty dislike the notion of measuring what students have learned and some find the idea that surveys are more meaningful than grades insulting. It has been compared with the idea of faculty being or not being able to receive tenure based on the number of publications they have. Other faculty believes measurement is useful as long as it is not at the expense of more important issues (McMurtrie, 2000).

Stallings (2002) asserts that published statements by accreditation commissions and other influencers indicate that there is an increased emphasis being placed on accountability in learning as well as the outcomes of the educational process. One such agency is the Commission on Colleges of the Southern Association of Colleges and Schools (SACS). SACS is tasked with accrediting institutions of higher education. They ensure that institutions meet established regional standards. At the heart of the Commission’s philosophy of accreditation is the concept of institutional effectiveness. Institutions must demonstrate how well they fulfill their purpose and must engage in an ongoing quest for quality. They must provide documentation of the quality and effectiveness of their programs. Conducting self-studies to evaluate established standards of quality and the strength of the institution and to determine that educational goals are being met is a way of addressing the issue of effectiveness. Institutions are expected to use their purpose statement as the basis of planning and evaluation, to use a variety of assessment methods, and to demonstrate use of the results to improve educational programs. Additionally, both quantitative and qualitative data that demonstrate student achievement should be used when evaluating an academic program (“Criteria for Accreditation,” 1998).

Purpose

Through external accreditation review processes, university self-study assessments, and student reports of self-learning, it is evident that student learning in the traditional face-to-face setting is demonstrated and validated through research. However, as more and more institutions of higher education are transcending the traditional boundaries to offer courses as well as entire degree programs on line, the issue of distance education effectiveness has spawned vigorous debates. Thus, this study will seek to identify multiple assessment strategies that can be used to evaluate the effectiveness of distance education courses within a degree program.

Review of Literature

McMurtrie (2000) identifies one common thread among accreditors—they are all revising or developing better ways for colleges to focus on and measure student learning. These accrediting agencies included the Middle States Association of Colleges and Schools, New England Association of Schools and Colleges, North Central Association of Colleges and Schools, Northwest Association of Schools and Colleges, Southern Association of Colleges and Schools, and the Western Association of Schools and Colleges, Commission for Community and Junior Colleges. Traditionally, within the classroom, a variety of means are used by effective teachers to determine how well students are learning. They use tests, quizzes, exams, research papers and homework to formally evaluate student achievement and to assign grades. To informally evaluate learning, teachers ask questions, listen to comments and questions from students, and monitor facial and body expressions. Such evaluations allow teachers to adjust their teaching by slowing down when necessary to review material or to clarify questions, confusion, and misunderstandings. They also allow teachers to move forward when students exceed expectations. However, different challenges face teachers who must teach at a distance. Teachers are no longer surrounded by a familiar, traditional classroom. They lack contact with a homogeneous group of students as well as face-to-face feedback such as student questions, body language, facial expressions, and comments. Thus, distance educators must not only formally evaluate students, but must also use an informal approach to collect data that determines teaching effectiveness, clarity of course content, appropriateness of assignments, and student comfort with the methods of delivery of instruction (“Guide #4,” n.d.).

Carnevale (2001) notes that the difference between distance assessment and traditional classroom assessment is that distance education programs for the most part are geared toward those already in the workforce. Since this involves learning by doing, students complete projects to show that they can apply what they have learned in addition to understanding it. Being able to apply what has been learned is a focus of many assessment policies. He also notes that some universities have created their own assessment exams, and some buy them from organizations such as the Educational Testing Service and ACT. These assessment exams may take the form of essays, multiple-choice questions, or projects whereby students must demonstrate their knowledge. A project might include simulations of a real world job. For example, students enrolled in information technology or network administration have to plan computer networks for small companies. The plan is presented in a five-page essay detailing financial and technical requirements for implementing the plan, similar to what an employee would present to the manager of a company. Students enrolled in a business science program have to create a marketing plan for a small company. In a five-page essay, students specify how a company can improve its contact with customers. Such projects are a part of the students’ portfolio requirement. The portfolios are reviewed to assess whether students have demonstrated their knowledge and skills to earn a degree. To ensure that students are able to work collaboratively once they enter the workforce, some assessment strategies include having students work in a collaborative fashion to create a product instead of just learning by lecture. In addition, the University of Phoenix expanded the assessment practices of its face-to-face program to its fast-growing distance education program. Both traditional and on-line students take the same 100 multiple-choice question tests at the beginning of their academic program and at the end. The data is then analyzed to see how much students have learned from the program. Further, projects are given that assess students’ critical thinking skills, communication skills, as well as cognitive abilities.

THE STUDY

Population
The study focuses on a Business, Career, and Technical Education Department within a College of Education at a large doctoral university in eastern North Carolina. The department offers 1) undergraduate teacher education programs: BSBE in Business Education, BSBE in Marketing Education, and BSBE in Business and Marketing Education; 2) undergraduate non-teaching programs: BSBE in Information Technologies and Information Technologies (Community College 2+2 Community Partnership); and 3) graduate programs: MAEd in Business Education and Marketing Education, and an MS in Information Technologies. At the time this study was conducted, the department also offered a degree in family and consumer sciences education. This program has since relocated to the Human and Environmental Sciences Department.

Methodology
To evaluate the effectiveness of the distance education program in information technologies, comparisons of three courses being taught simultaneously on line and in the classroom were made to see if significant differences existed in student outcomes assessments. Students within these courses were assigned specific tasks to be evaluated. The criterion established for these courses was reviewed by two or more independent raters. The classes included financial information systems II, business communications, and programming with Microsoft visual basic. In addition, most of the programs within the department are offered in the classroom as well as on line, many of them simultaneously. Thus, the same assessment measures used for SACS for the traditional classroom were equated with the distance education classes for those in the field of Family and Consumer Sciences, Business Education, the BE/ME program, the MAEd program in Technical teaching, and Information Technologies.

ASSESSMENT OF CLASSES

Financial Information Systems II Class
Traditional and distance education accounting students were required to produce an accounting simulation using Peachtree Accounting software. Ninety percent of the students were required to attain a mean rating of 80% or higher on a 100-point rating scale on an audit exam of the simulation. Two independent raters evaluated the 1) Accuracy of Transaction Entries, 2) Accuracy of Printed Formal Reports, 3) Completion of all Reports, and 4) Accuracy of the Audit Exam. Below are the results for the traditional campus section and for the distance section. The scale used for this assessment was: Scale: 1 = poor 2= fair 3 = good 4 = very good 5 = excellent.

Business Communication Class
Business communication students were required to work collaboratively in groups of three to produce a research paper on international business communication that portrayed their expertise in the researching of international business communications as a final project. Criteria for success was that 90% of the students would attain a mean rating of 4.0 or higher on a 5 point rating scale when the papers were assessed by two independent raters.

Research papers were evaluated for content, use of APA writing style, mechanics, grammar, and punctuation. The results revealed the following:

Programming with Microsoft Visual Basic Class
Students in a basic programming for business applications course had to produce a computer software program that portrayed their expertise in the design and programming of a visual basic computer program. Criteria for success were that 90% of the students would attain a mean rating of 4.0 or higher on a 5 point rating scale.

ASSESSMENT OF PROGRAMS

There are certain requirements that all students who graduate from the program must meet, whether taking courses by distance or traditionally. The same standard formal assessment measures used in traditional classes to satisfy SACS requirements were also used in distance education classes. Examples include items such as:

All students, traditional and distance were required to meet the same set of criteria as that listed above. Samples of this information are posted and can be found at:

http://www.ecu.edu/pir/ARBS/AcademicUnits/Business%20Education,%20B.S.B.E.htm.

Finally, the Family and Consumer Sciences program also had specific criteria identified for both traditional and distance education students. First, students in this program were required to produce a traditional paper 1040 and a computer generated 1040 income tax return. A criterion for success was that 90% of the students would successfully pass the audit component on their first attempt. In each of the cases above, both traditional and distance students met the criteria for success. There was no significant difference between the two groups. Second, the students had to develop an early experience manual that documented 16 hours of clinical experience in the FACs program area. The criterion for success was that 95% of the manuals would contain observation logs, reflections, and classroom analysis when evaluated by a team consisting of the university professor, the clinical teacher, and a teacher educator from the BVTE Department. Both groups met the criteria for success. Third, students were required to participate in a minimum of one one-day professional development activity. Criteria for success was that 95% of the students would attend and document the professional activities participated in through a reflection when evaluated by a team consisting of three faculty members in the BCTE department. Both groups met the criteria for success.

CONCLUSION

The assessment measures provided identified multiple assessment strategies that can be used to evaluate the effectiveness of distance education courses within a degree program. These strategies included projects, research papers, portfolios, qualitative, and quantitative data. Results of these measures revealed no significant difference in achievement between traditional campus and distance education students.

REFERENCES

Mc Murtrie, B. (2000). Accreditors revamp policies to stress student learning. Chronicle of Higher Education, 46(44), pA29, 3p, 2c.

Carneval, D. (2001). Assessment takes center stage in online learning. Chronicle of Higher Education, 47(31) pA43, 3p, 1c.

Criteria for Accreditation (1998). Retrieved November 18, 2003 from .
http://www.sacscoc.org
Guide #4 Evaluation for Distance Educators (n.d.). Retrieved May 22, 2002, from University of Idaho, College of Engineering Web site: http://www.uidaho.edu/evo/dist4.html

Stallings, D. (2002). Measuring success in the virtual university. Journal of Academic Librarianship, 28(1/2) p.47, 7p.

Biodata of author:

Dr. Shelia Tucker is an assistant professor in the Department of Business, Career, and Technical Education at East Carolina University (ECU), Greenville, North Carolina. She received her PhD in vocational-technical education with a specialization in business education in 1998 from Virginia Tech. Employed at ECU since August 1998, her interests are in effective delivery of distance education as well as the integration of cutting edge technology to facilitate teaching and learning. tuckers@mail.ecu.edu

Dr. Elizabeth Hodge is an assistant professor in the Department of Business, Career and Technical Education at East Carolina University (ECU), Greenville, North Carolina. She received her Ph.D. in educational leadership with an emphasis on higher education administration technology and policy from Florida State University in 2000. Employed at ECU since 2000, her interests are in effective delivery of distance education as well as the integration of cutting edge technology to facilitate teaching and learning.