Case Story: Open Comment
Case Story: Open Comment
| Summary | Open Comment | |||
|---|---|---|---|---|
| Group / workshop | Formative e-Assessment | Status | seed | |
| Project details... | ||||
Situation
What was the setting in which this case study occurred?
Multiple choice questions had been used by the Arts Faculty at the Open University to support student learning through a course but the academics believed this type of electronic testing did not suit their discipline. They wanted to ask more difficult questions and for students to answer questions in their own words, i.e. “free text entry”, which in turn would be automatically marked. The PVC funded this project initially for six months in 2006 to establish proof of concept. A further tranche of money was funded towards the end of 2007 to build a system that could be used in earnest by the Faculty before it would be used on the course. The Faculty are now starting to use the system but a delay of 12 months occurred between completion of the system and use by Faculty members because the leading academic was on study leave and the system needed to be rebuilt on the OU servers as it was originally maintained by the Robert Gordon University. The clients were the History Department of the Arts Faculty at the OU. The primary interaction with the History tutors was to understand the nature of the questions asked and the types of feedback given to the student so that this could be modelled in the computer system. Once a prototype was constructed the interface and content would be interactively tested with them.Task
What was the problem to be solved, or the intended effect?
This work coincided with the OU’s adoption of Moodle as its VLE and so any free text e-assessment system needed to be supported by Moodle. Therefore the specific objective of the project was to construct some simple tools in the form of Moodle extensions that allow a Moodle author to ask free-text response questions that can provide a degree of interactive formative feedback to students. In parallel with this was the aim to begin to develop a methodology for constructing such questions and their feedback effectively, together with techniques for constructing decision rules for giving feedback. Open Comment is a formative feedback technology designed to be integrated in the Moodle virtual learning environment. Put simply, it provides a simple system allowing questions to be written in Moodle, and for students’ free text responses to these questions to be analysed and used to provide individually customised formative feedback. This question type was designed for use by History Undergraduate students on the Arts Foundation course of the Open University. It was a deliberate and easy decision to separate the feedback engine from the VLE as a web service. This is in keeping with JISC’s emphasis on service-oriented architectures. However, generating feedback is computationally intensive, and Moodle is implemented in a language that is not suited to computationally intensive processing. Using this approach allows the load to be balanced, with the VLE running on one set of servers, and feedback generation on separate systems if required. A second benefit of this is that only the presentation aspects of the system need to be adapted to additional VLEs. Open Comment has been developed as an open source system, and consists of the following components: · A Java-based feedback system· A web service shell· A Moodle-based question type· A graphical interface for testing· A forms-based editing toolActions
What was done to fulfil the task?
The feedback model operates by and large through a sequential set of rules identifying sources of evidence within the student’s response, and escalating in level of analysis, in some sense following Anderson, Krathwold, and Bloom’s (2000) revised taxonomy of educational objectives. Importantly, also, there is a strong causal element to many of the rules. These rules are implemented in a bespoke feedback engine within Open Comment – by and large, all the other components are only there to make it accessible in a usable form, through a VLE or through an interactive interface. Much of the feedback model is implemented in JavaScript rules, which make the bridging inferences between the levels. Simple errors of omission or commission can be immediately added to the response; otherwise, the analysis passes on to more detailed feedback on later stages. Each question is analysed using a script in a configuration file, allowing many questions to be configured and handled from the same main feedback engine. Each question will typically provide its own configuration file, although this is not always necessary, as in some cases several questions may be closely related, and share aspects of inference about appropriate feedback.Results
What happened? Was is a success? What contributed to the outcomes?
Questions that suit this system well are those that exhibit a causal model. Three types were suitable. These were: 1. Analysis of statistics, usually presented to the student as a table 2. Comprehension of a set text 3. Identifying similarities and differences for a given event The work of Dweck and her colleagues inspired the feedback model where effort was recognised. The first demonstration system was received favourably by Arts Faculty staff who have now become more aware of both the potential and limitations of automated systems based on free text responses. Lessons have been learned about the type of feedback that instructors think would be most useful. In particular, we have found that it appears to be worth distinguishing two main classes of feedback. These being: · Specific to the question · Generic for Arts-style questions The system can deal with an answer submitted by the student such as “no idea”. A response would be generated rather like this: “maybe you are a bit confused by the question. It may be helpful to remember you are not being asked directly about the causes of X but why X has been .....” Please see attached screen shots. A certain degree of feedback to students on free text answers can be usefully generated, but cannot with the current state of the art, replace detailed feedback from a qualified academic. The benefit to the students is that helpful feedback can be given almost instantaneously. This should encourage more rapid progress and build student confidence. The benefit to the course tutor is that more off-the-point responses should be identified by the system so that the tutor’s attention can be focused on more substantial issues that are pertinent to the students.Lessons Learned
What did you learn from the experience?
1. The bulk of the time on the project was used in analysing the tutor feedback to students on the three different types of causal questions. This part of the project was easier to complete because of the detailed written feedback that is given automatically by OU tutors to their students. 2. An ethnographic study was also undertaken by the Educational Technologist and the Programmer i.e. they both became History students and answered the questions. This part of the project revealed that maximum marks can be gained only when the student understands “the rules of the game”. Once the researchers had undertaken a knowledge elicitation session with the History tutor, they were able to increase their grades from a C to an A because they were aware of these “rules of the game”. 3. Incorporating feedback about “the rules of the game” we felt should be included in the feedback, perhaps with a separate button that could be turned on or off by the student. Surprisingly this idea is being resisted, not by OU tutors, but by others from a number of universities where the system has been demonstrated. 4. Multiple disciplinary teams are needed to conduct this type of work. 5. Editing tool that tutors can use directly is an essential component of this system. 6. A pedagogical model of feedback has been developed which is now open to test. We are waiting on findings from student usage.Licensing
This work is licenced under a Creative Commons Licence.
