Art 1
[ Home ] [ Up ] [ Alt L-T Experiences ] [ Alt Assessment ] [ Course Programme ] [ Lunch Hours 2006 ] [ Workshops 2006 ] [ Grids & Rubrics ] [ Problem Solving Techniques ] [ OLESP ] [ NQF&SAQA ] [ Learning Programme Development ] [ Teaching Portfolio ] [ Article Writing ] [ Workshop Programme ] [ New Lecurer Sem 2003 ] [ Assessor Training ] [ ProTechnida ] [ CTP Documments ]

ABSTRACT

Internet-based distance education is becoming a powerful player in the field of education. Testing mechanisms in this field are lacking at the present time, partly due to a lack of a model around which to implement testing and evaluation mechanisms. This paper proposes a generic model for an evaluation mechanism and shows the effectiveness of the model in an operational collaborative learning package.

In the modern tertiary institution, distance education is becoming more important as the focus of education changes. Internet-based distance education is a rapidly expanding field and one that provides the greatest opportunities for distance education in the global arena. The primary area in which Internet-based distance education is lacking is that of testing and evaluating students in an effective manner. This paper investigates the requirements for an effective evaluation mechanism and proposes a generic model for use in evaluation systems.

The focus of innovation in tuition in the modern institution is moving away from traditional, residential oriented course methods and towards lifelong learning where persons seek knowledge as and when they require it. Such methods of learning often preclude the attendance of campus-based classes and, as a direct result of this trend, distance education in general, and Internet-based distance education in particular, is becoming more important to institutions (PSU Task Force on Distance Education, 1992).

One of the more effective mechanisms for delivering distance education over the Internet is a specific type of network known as an Asynchronous Learning Network (ALN). These networks fulfil most aspects of the modern campus classroom. However, ALNs are designed to work asynchronously and thus have to ensure that students can work at anytime, from anywhere, and still gain the full experience of the campus-based classroom (Mayadas, 1997 p2).

A primary feature of modern education is the trend towards collaborative learning within the classroom, leading to a greater involvement of students in the education process. In the online distance arena, ALNs are the most viable solution for conducting collaboration. Features such as discussion forums, information sharing and peer evaluation mechanisms facilitate time- and location-independent collaboration.

The main area in which ALNs, and most online educational environments, show a large deficiency compared to the traditional classroom is that of evaluation of students. In traditional education, students can be evaluated by many different forms and give the instructor an accurate measurement of the knowledge of the student. However, most of these methods are not available in the online classroom, and those that have been implemented are usually proprietary systems that do not inter-operate with other systems (Sieborger and Macintosh, 1998, p5-6).

As a result of the lack of online assessment methods, assessment tends to fall back on the traditional methods which reduces the efficiency of distance education. There is a clear and present need for a generic, inter-operable mechanism for conducting tests and evaluations in online environments. This paper proposes a model for such a mechanism and shows that the model is a valid solution for the problem by implementing a prototype based on the model.

An Asynchronous Learning Network, or ALN, is a concept that takes all aspects of a physical classroom and attempts to emulate those aspects in an online environment. Mostly, ALNs take the form of custom designed Internet applications, allowing any student to take part in courses from anywhere in the world, the only constraint being the availability of an Internet connection (Mayadas, 1999 pp1-16).

In order to do this an ALN requires a number of common features that present the student with a true online classroom. Simply, these features must endeavour to provide the facilities and experiences that are found in the physical classroom. In addition, these facilities must be provided in such a way that the members of the ALN can use them in both synchronous and asynchronous manners, i.e. the implementations must be time-independent (Bourne, 1998 pp 71-72).

Firstly there is a mechanism for students and instructors to share resources, be it web resources, course notes or software. In an ALN, this is usually implemented as online libraries where members of the network can access shared resources via their browser software and also upload new resources.

Secondly, students and instructors must be able to communicate with one another. This extends beyond the concept of e-mail and bulletin boards and is intended to facilitate collaborative learning within the group. While e-mail, chat rooms or live conference facilities can fulfil this requirement, enhancements to these are needed to provide true asynchronicity within the group. Internet newsgroup style discussion forums where messages are stored permanently, come closest to fulfilling such roles.

Lastly, evaluation of students within the ALN is vital. Understanding how much a student has learned and evaluating the understanding of learned knowledge is one of the most important aspects of any education process. Evaluation will be discussed below in much more detail. It is interesting to note that some researchers include features such as live chat and video conferencing as vital parts of an ALN, despite the fact that such facilities break the asynchronous nature of the concept of an ALN.

Having identified the common features of an ALN, the application of these features in providing an online environment conducive to collaborative learning is the next stage.

Collaborative learning is the process by which students learn by sharing the learning process with one another. This sharing can take the form of group work, discussions and debates, peer review and information sharing. In the physical classroom, this sharing takes place automatically, students discuss with instructors and one another, criticise others’ comments and take part in group work as a team member (Hiltz, 1997 p3).

In distance education, these activities are missing. Traditionally there is no real contact between the students and only minimal contact between a student and an instructor. However, an ALN can provide such facilities and the basic ALN designs implement many methods whereby interaction between student and instructor is possible.

It has been shown that by actively encouraging students to participate in discussion forums in online environments, the same atmosphere of knowledge sharing and learning that is present in the physical classroom can be obtained. However, as in a physical classroom, the discussions have to be controlled and guided along the correct path (Wegner, Holloway & Garton 1999 pp 98-106).

In current ALNs, one of the facilities that are lacking is a flexible evaluation system. An integrated evaluation system allows students to take tests and examinations online, as well as practising assignments without leaving the ALN environment. The evaluation system is also used in supporting collaborative learning in an ALN.

One of the main elements of the collaborative class is peer evaluation. This method of evaluation uses both student and instructor input when grading the performance of a student in group work. Group members will give one another various marks for areas of work, outlined by the instructor, and these marks are combined with marks given by the instructor into a final grade for each student. This is a time-consuming process that involves repetitive work better suited for a computerised evaluation system.

Providing evaluation and testing in a computer based environment has always proved troublesome. There is a trade-off between the depth and complexity of questions and the amount of automation that can be provided, and many current implementations focus on fully automated testing systems. These automated systems are limited to simple forms of testing such as multiple choice and true/false questions (Ron, Kwok and Jain, 1999).

The direction that the design of computer-based examination systems has taken in the last few years has concentrated on secure and accurate mass testing systems. These systems are based on simple question types and use massive question databases to provide security and unpredictability in testing. While these tests are suited for some scenarios, they are unsuitable for evaluating learning and understanding at a tertiary level (Bennet, 1998).

Much research has been carried out in the field of evaluation at tertiary levels, with the consensus of opinion being that complex testing is required to accurately quantify the level of understanding that a student has gained during study. These tests include structured answers where the student has to show how a decision is made, essay or paragraph questions involving explanation and application of knowledge and practical application of knowledge in laboratory or work-related environments.

In order for a evaluation system in a distance learning environment to be fully applicable to the tertiary level, the system must encompass most, if not all, of the previously discussed testing elements. Most of the currently available systems are not able to reach the required level for proper evaluation; the distance student is often required to attend traditional tests and examinations on a campus. This precludes true distance education and narrows the scope of such programs (Ron et al, 1999 pp109-125).

It is obvious that a system that can provide these features in a distance education environment will be a great aid to the future of distance education and to education in general. While the lack of automated marking of some question types is regarded as a drawback in modern systems, and the fact that there are issues outstanding with regards to the identity of the person on the remote computer, the provision for taking and grading of tests via an online system is a prime motivation for the model proposed in this paper.

As discussed in the previous section, applications for evaluating students in an online environment are in the early stages of development and are not advanced enough to perform evaluations at a level required for modern education. Additionally, present solutions tend to be monolithic entities designed to work within one environment and with little or no future extensibility in mind.

It is clear that the feature set of an evaluation system will change over time and to accommodate this, the system needs to be easily adaptable. To accommodate this level of flexibility, a monolithic structure is not feasible. A highly modular design provides the required extensibility and future proofing that is required of such a system and separating the core functions in individual modules allows for the system to be enhanced or extended as is necessary.

The proposed model consists of a number of co-dependent modules, each providing certain services to the model and requiring other services. Figure 1 is a block diagram of the model showing the basic module structure.

The complete evaluation process will be online via a WWW browser, with all sections of the session being presented as an HTML document. The lecturer then retrieves each student's answer script via his browser and grades the answers before submitting the graded paper to the server. The final marks are then stored on the server in a database.

Each module will be discussed below, giving detailed information on the functionality, security concerns and generic operations of each module.

The Core Application Module (CAM) is the main flow of the model, supplying control structures and functions to all other modules. It implements an XML parser that processes the evaluation script files. Inter-module security functions are implemented to protect module workspaces and include protection for temporary space used by the system for each evaluation.

Figure 1: Overall Architecture of the Model

XML has been chosen as the language in which evaluations are scripted as it is an open standard that is specifically designed for defining new languages. The open nature of the standards ensure that XML, and therefore the evaluation language, will remain public and compatible with any implementation of the model, i.e. any test script will work with any evaluation software.

The model only specifies that a source XML document be used. It should not be expected of instructors to know XML at all. An authoring tool should rather be designed to generate the XML files automatically. This tool is left up to the implementers.

The CAM does not implement evaluation type dependent functions, database access functions and high-level security functions. These have been extracted to separate modules to allow for further independence of the design. This also allows core functions to be changed without affecting the rest of the application.

The CAM also manages the initial loading and management of the modules, utilising any existing dynamic module loading mechanisms provided by the underlying computer, as well as the interface to the WWW server that interacts with the user on behalf of the evaluation system.

The model specifies that certain modules, question modules and evaluation modules, are loaded only when needed, demand-loaded modules, and unloaded when the features are no longer required. All other modules, database access and security, are loaded once and remain in memory as resident modules. Such a policy has two immediate advantages: Firstly, memory usage is reduced by only having required modules present and secondly, functions exported by each of the question modules can have the same names without causing clashes.

The CAM is the core of the model, providing common services to all other modules that plug into the model. The types of module that plug into the CAM will now be discussed.

The Database Module (DM) provides connectivity to a database and allows the modules to utilise the database. The DM is intended to provide implementation-independent database access to the modules, hiding the nature of the database from the modules. The only requirement is that the database must be a relational database.

This is done in the DM by implementing a series of abstraction functions that call the native functions of the database in turn. Thus a module using a function need not have to worry about what database is being accessed. All it does know is that a query is performed and that data is returned in a standard format. The DM has to reformat the data into the standard format and present it to the requesting module.

The security module (SM) is responsible for the high-level security functions of the model. It limits access to the test to a particular time as scheduled by the lecturer. During the test it keeps track of the students’ progress. Access control is done via roles to which rights have been assigned and limitations are used to perform rapid checks as to the student's rights at any given time. The security information is given in the XML script file and loaded by the XML parser into the central data structures for access by each module.

The Evaluation Type Module (ETM) is responsible for handling the upper level functions of an evaluation. Each module is designed to handle a different form of evaluation, for example a class test, an examination or peer evaluation.

In order to accomplish this, an ETM must provide services for generating form headers, collating submitted results, and handling of grading and final mark calculations. The ETM automatically hands each question off to the respective question module during each stage of the evaluation process.

The ETM is responsible for generating a suitable HTML document presenting the evaluation to the student. This document is based on the XML script for the evaluation in question. It does this using an evaluation generator, a function that interprets the XML from the parser and calls the relevant sub-modules to generate the page.

The generator is required to provide the HTML skeleton for the document, including any style and formating that the site deems necessary, custom HTML form tags, session management and client-side scripts that may be included for extra functionality. In addition to this, the generator in the ETM must frame any HTML generated by the question modules to fit the HTML objects into the layout of the final document.

A major part of any evaluation process is the statistical information gleaned from the testing process itself. While information about grade spreads and similar facets can be obtained from the stored results at the end of the testing process, information on time spans taken by students per question can only be logged at testing time. Client-side code can log timestamps when an element is used, giving a reasonably accurate timeline.

When a student submits the answers to the server, the answers and related information are stored in the database for future reference, identified by unique codes based on the identity of the student and the specific question.

The test grading engine forms the second core region of the ETM. One of the fundamentals of this model is the separation of grading for different question types and the grading engine performs a similar function to that of the test generation engine, generating the HTML document and handing the grading of each individual question to the relevant question module. The generated document is equivalent to the paper script handed in by the student in a conventional evaluation.

The final stage in the evaluation process is the finalising of the grade for each student. After the lecturer has reviewed the answers given by the student and assigned marks for those questions that require them, the ETM must take the marks given, weight them according to specifications in the XML file, and determine a final grade for the student. The final result is stored in the grade database of the environment and made available to the student via the usual grading channels.

The grade calculator is also responsible for weighting groups of questions. If the author of the XML script uses the grouping features and supplies a weight for each group, the calculator must include this into the calculation process of the final grade.

The question module (QM) forms the most important part of the model. It is these modules that provide the high degree of flexibility and extensibility that is the hallmark of this model. A QM can implement any form of question, for example multiple-choice, single answer, paragraph/essay and equations, as well as providing a mechanism for grading the answer either automatically or by referring the answer to the lecturer.

A QM does not encapsulate a process like the CAM or the ETM does. It provides a set of standard functions to the ETM that handle all facets of the question. In fact, a QM can be seen as a set of six individual modules combined into a single transport package.

Using such a mechanism allows both entire modules and functions to be replaced as and when new technology is developed. For example, at the moment marking of a paragraph question is handed to the lecturer to do, but if a mechanism or application for grading a natural language passage is written, the relevant functions can be replaced without causing problems with the rest of the module.

The question generator takes the XML data for the specific question and creates a small block of HTML code that represents the question. For example a multiple-choice module may take in XML specifying the prompt for the question and the options. It will then generate a section of HTML code that will allow the student to answer the question. This is shown below in Figure 2 with sample screenshots from the prototype that was implemented.

Figure 2 - Stages in Generating a Question

Adding other features to a question, such as multimedia clips, external links and images, is accomplished by adding definition tags to the XML and modifying the generator function to output the correct HTML for the features. Questions can become more interactive and closer to real life than is possible with current systems, both conventional and online.

The QM also retrieves the answers and timing information that the student submits and writes them to the database. There is no limit to what may constitute an answer, for example, a word, letter, equation, paragraph or image, and again this allows for future expansion and flexibility.

The lecturer grades the answer script of the student online. In order to reduce the workload of the lecturer and automate the evaluation process, any questions that can be graded automatically, are handled by the grading function. Since a question can be either graded automatically or manually, e.g. a paragraph question, this function is not concerned with either the weighting of the question or the final grade for the evaluation.

Like the question generator, the grading function generates a block of HTML code, encapsulating the question, the student's answer, the correct answer if available and a text input field for the grade. If the question is marked automatically, the grade will be placed in the field.

As in the question generator, any feature desired can be added by extending the question tags in the XML document and handling the extended features in the function. For example, multiple grade fields may be required if the question has several sections.

The final step of the grading process occurs when the lecturer submits the results to the server. The submission function retrieves the grade given, applies any weighting or statistical function that may be required and saves the result with the student's answer in the permanent database. It is vital that the question modules generate one, and only one value that is normalised to 100%. This is required so that the ETM can simply add up all the results for a student and determine a percentage result.

The QM system is a highly flexible system for question management within the model. It can be easily extended to add new question types or to provide a new mechanism for handling an existing question type. The model is not concerned with what types of question can be implemented; only the mechanism by which it can be done.

In order to evaluate the proposed model, a prototype was implemented according to the model. The prototype was integrated into an existing ALN environment called IESP, Internet Education Support Package, created by the author and evaluated as to the effectiveness and the integration levels. The PERL programming language was used as the development language for the prototype, both for the flexibility of the language and the speed with which changes can be made.

Two forms of evaluation were implemented, a normal examination and a peer evaluation mechanism. Within these high level forms, a number of question types were tested, namely multiple choice, true-false, paragraph and group rating question types. The group rating type allows the student to rate the rest of the group members for each question.

Two problems were noted during implementation. Firstly, the differences between evaluations were not sufficient to warrant the creation of separate modules. Switching between the two types within the CAM was used instead. Secondly, as PERL does not support function references, the functions were referenced via the complete name within the module hierachy. These problems do not break the model as proposed

Figures 3 and 4 show a sample test from the student’s and instructor’s perspective respectively. Effectiveness tests were conducted with various forms of examinations and proved successful. The testing and grading interfaces worked as expected from the design. While some minor interface issues arose, these are simply HTML problems and are easily solved.

Figure 3: Student view of test page

Figure 4: Instruction view of test script

As discussed in this paper, there is a clear move towards distance education in the modern tertiary institution. Using the Internet for delivering distance education to geographically distant students is a proven field of application and asynchronous learning networks, ALNs, are a proven delivery mechanism.

Evaluation of students in both physical and online classrooms is a vital area of research and while great strides have been made in the physical environment, the same advances are not arriving in online classrooms. The main reason for this is the desire for complete automation in the testing process. A need for a viable testing mechanism was identified.

This paper proposed a generic model for an evaluation engine that modularises the evaluation process and implements physically separate modules for each form of question and evaluation. A trade-off was achieved in the model with regards to the testing and grading processes and grading is done via a combination of automated and instructor grading. The model was tested by implementing a prototype and was proved to be a valid solution to the problem outlined.

Bennet, R R. (1998). Reinventing Assessment: Speculations on the Future of Large-Scale Educational Testing. ETS Policy Information Report. Retrieved from the Web 7/4/2000 http://www.ets.org/research/pic/bennett.html

Bourne, J R. (1998). Net-Learning: Strategies for On-Campus and Off-Campus Network-Enabled Learning. Journal of Asynchronous Learning Networks. 2(2) pp 70-88

Hiltz S R. (1997). Impacts of college-level courses via Asynchronous Learning Networks: Some Preliminary Results. Journal of Asynchronous Learning Networks. 1(2) pp 1-19

Mayadas, F. (1997). Asynchronous Learning Networks: A Sloan Foundation Perspective. Journal of Asynchronous Learning Networks. 1(1) pp 1-16

PSU Task Force On Distance Education. (1992). The Report Of Task Force On Distance Education Retrieved from the Web 4/4/2000 http://www.outreach.psu.edu/de/de_tf.html

Ron, C W, Kwok, W & Jain Ma. (1999). Use Of A Group Support System For Collaborative Assessment. Computer & Education 32 pp 109-125

Sieborger, R. & Macintosh, H. (1998). Transforming Assessment: A Guide For South African Teachers. Cape Town: Juta

Wegner, S B. Holloway, K C. & Garton, E M. (1999). The Effects of Internet-Based Instruction on Student Learning. Journal of Asynchronous Learning Networks 3(2) pp 98-106