Changes for document OpenComment
From version 17.1
To version 23.1
edited by john gray
on 2009/03/04 16:18
on 2009/03/04 16:18
Change comment: Updated class property name
| Metadata changes | ||
|---|---|---|
| Property | Previous value | New value |
| Parent | Cases.WebHome | |
| Document author | john gray | |
| Default language | en | |
| Content changes |
|---|
#includeForm("Cases.CaseClassSheet") |
| Comment changes | ||
|---|---|---|
| Property | Previous value | New value |
| Comment number 0 added | ||
| Author | Yishay Mor | |
| Date | 10/10/2008 17:28:37 | |
| Comment content | excellent task description! The only bit I miss (perhaps pertaining to situation more than task) is - was this designed as a generic tool, or did you have a specific course in mind? | |
| Comment number 1 added | ||
| Author | Yishay Mor | |
| Date | 10/10/2008 17:32:34 | |
| Comment content | The situation includes theoretical background which would suit an introduction section of a paper, but does not fit well here. The situation needs to provide a clear context for the task and the actions - did this work emerge out of a specific teaching setting, or as a commissioned work? Who were your "clients" and your "users" and what was the nature your interaction with them? What was the time frame for development? | |
| Comment number 2 added | ||
| Author | Yishay Mor | |
| Date | 10/10/2008 17:43:35 | |
| Comment content | much of the "actions" section belongs in the "task" description, as it tells us what your product is, not how it came to be. Actions should describe the steps you took to make this happen, the challenges you encountered along the way and how you overcame them. Likewise, much of the "results" should move to the "actions", as they describe the system implementation rather than its effect. The results should provide detail on your observations of the systems performance with respect to the task definition, along with any unexpected outcomes. | |
| Comment number 3 added | ||
| Author | Denise Whitelock | |
| Date | 21/10/2008 13:26:26 | |
| Comment content | Yishay, Further to your suggestions above, I have rewritten the text. I am also sending you by email some further screen shots to explain the system. | |
| Object changes | ||
|---|---|---|
| Property | Previous value | New value |
| Object number 0 of type Cases.CaseClass added | ||
| Owner | XWiki.DeniseWhitelock | |
| LegalRights | Yes | |
| Situation | {style:type=span|color=#000000}Multiple choice questions had been used by the Arts Faculty at the Open University to support student learning through a course but the academics believed this type of electronic testing did not suit their discipline. They wanted to ask more difficult questions and for students to answer questions in their own words, i.e. “free text entry”, which in turn would be automatically marked.{style}{style:type=span} {style}{style:type=span|color=#000000}The PVC funded this project initially for six months in 2006 to establish proof of concept.{style}{style:type=span} {style}{style:type=span|color=#000000}A further tranche of money was funded towards the end of 2007 to build a system that could be used in earnest by the Faculty before it would be used on the course.{style}{style:type=span} {style}{style:type=span|color=#000000}The Faculty are now starting to use the system but a delay of 12 months occurred between completion of the system and use by Faculty members because the leading academic was on study leave and the system needed to be rebuilt on the OU servers as it was originally maintained by the Robert Gordon University.{style}{style:type=span|color=#000000} {style}{style:type=span|color=#000000}The clients were the History Department of the Arts Faculty at the OU. The primary interaction with the History tutors was to understand the nature of the questions asked and the types of feedback given to the student so that this could be modelled in the computer system. Once a prototype was constructed the interface and content would be interactively tested with them.{style}{style:type=span} {style} | |
| Group | Formative e-Assessment | |
| Lastedited | 06/05/2008 14:10:50 | |
| Summary | Open Comment | |
| Name | Open Comment | |
| Status | seed | |
| Created | 06/05/2008 14:10:50 | |
| License | CC-BY-NC-SA (recommended) | |
| Task | {style:type=span|color=#000000}This work coincided with the OU’s adoption of Moodle as its VLE and so any free text e-assessment system needed to be supported by Moodle.{style}{style:type=span} {style}{style:type=span|color=#000000}Therefore the specific objective of the project was to construct some simple tools in the form of Moodle extensions that allow a Moodle author to ask free-text response questions that can provide a degree of interactive formative feedback to students.{style}{style:type=span} {style}{style:type=span|color=#000000}In parallel with this was the aim to begin to develop a methodology for constructing such questions and their feedback effectively, together with techniques for constructing decision rules for giving feedback.{style}{style:type=span|color=#000000} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}Open Comment is a formative feedback technology designed to be integrated in the Moodle virtual learning environment. Put simply, it provides a simple system allowing questions to be written in Moodle, and for students’ free text responses to these questions to be analysed and used to provide individually customised formative feedback.{style}{style:type=span} {style}{style:type=span} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}This question type was designed for use by History Undergraduate students on the Arts Foundation course of the Open University.{style}{style:type=span|color=#000000} {style}{style:type=span|color=#000000}It was a deliberate and easy decision to separate the feedback engine from the VLE as a web service. This is in keeping with JISC’s emphasis on service-oriented architectures. However, generating feedback is computationally intensive, and Moodle is implemented in a language that is not suited to computationally intensive processing. Using this approach allows the load to be balanced, with the VLE running on one set of servers, and feedback generation on separate systems if required. A second benefit of this is that only the presentation aspects of the system need to be adapted to additional VLEs.{style}{style:type=span|color=#000000} {style}{style:type=span|color=#000000}Open Comment has been developed as an open source system, and consists of the following components:{style}{style:type=span|color=#000000} {style}{style:type=span|font-family=Symbol}·{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}A Java-based feedback system{style}{style:type=span|font-family=Symbol}·{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}A web service shell{style}{style:type=span|font-family=Symbol}·{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}A Moodle-based question type{style}{style:type=span|font-family=Symbol}·{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}A graphical interface for testing{style}{style:type=span|font-family=Symbol}·{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}A forms-based editing tool{style}{style:type=span|color=#000000} {style} | |
| Actions | {style:type=span|color=#000000}The feedback model operates by and large through a sequential set of rules identifying sources of evidence within the student’s response, and escalating in level of analysis, in some sense following Anderson, Krathwold, and Bloom’s (2000) revised taxonomy of educational objectives. Importantly, also, there is a strong causal element to many of the rules. These rules are implemented in a bespoke feedback engine within Open Comment – by and large, all the other components are only there to make it accessible in a usable form, through a VLE or through an interactive interface. {style}{style:type=span} {style} {style:type=span|color=#000000}Much of the feedback model is implemented in JavaScript rules, which make the bridging inferences between the levels. Simple errors of omission or commission can be immediately added to the response; otherwise, the analysis passes on to more detailed feedback on later stages. Each question is analysed using a script in a configuration file, allowing many questions to be configured and handled from the same main feedback engine. Each question will typically provide its own configuration file, although this is not always necessary, as in some cases several questions may be closely related, and share aspects of inference about appropriate feedback. {style} | |
| Results | {style:type=span|color=#000000}Questions that suit this system well are those that exhibit a causal model. Three types were suitable. These were:{style}{style:type=span|color=#000000} {style} {style:type=span}1.{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}Analysis of statistics, usually presented to the student as a table{style} {style:type=span}2.{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}Comprehension of a set text{style} {style:type=span}3.{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}Identifying similarities and differences for a given event{style}{style:type=span|color=#000000} {style} {style:type=span|color=#000000}The work of Dweck and her colleagues inspired the feedback model where effort was recognised.{style}{style:type=span|color=#000000} {style}{style:type=span|color=#000000}The first demonstration system was received favourably by Arts Faculty staff who have now become more aware of both the potential and limitations of automated systems based on free text responses.{style}{style:type=span} {style}{style:type=span|color=#000000}Lessons have been learned about the type of feedback that instructors think would be most useful.{style}{style:type=span} {style}{style:type=span|color=#000000}In particular, we have found that it appears to be worth distinguishing two main classes of feedback. These being:{style}{style:type=span|color=#000000} {style} {style:type=span|font-family=Symbol}·{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}Specific to the question{style} {style:type=span|font-family=Symbol}·{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}Generic for Arts-style questions{style}{style:type=span|color=#000000} {style} {style:type=span|color=#000000}The system can deal with an answer submitted by the student such as “no idea”.{style}{style:type=span} {style}{style:type=span|color=#000000}A response would be generated rather like this: “maybe you are a bit confused by the question. It may be helpful to remember you are not being asked directly about the causes of X but why X has been .....”{style}{style:type=span} {style}{style:type=span|color=#000000}Please see attached screen shots.{style}{style:type=span|color=#000000} {style}{style:type=span|color=#000000}A certain degree of feedback to students on free text answers can be usefully generated, but cannot with the current state of the art, replace detailed feedback from a qualified academic.{style}{style:type=span} {style}{style:type=span|color=#000000}The benefit to the students is that helpful feedback can be given almost instantaneously.{style}{style:type=span} {style}{style:type=span|color=#000000}This should encourage more rapid progress and build student confidence.{style}{style:type=span} {style}{style:type=span|color=#000000}The benefit to the course tutor is that more off-the-point responses should be identified by the system so that the tutor’s attention can be focused on more substantial issues that are pertinent to the students.{style}{style:type=span} {style}{style:type=span|color=#000000} {style}{style:type=span|color=#000000} {style} | |
| Lessonslearnt | {style:type=span|color=#000000} {style}{style:type=span}1.{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}The bulk of the time on the project was used in analysing the tutor feedback to students on the three different types of causal questions. This part of the project was easier to complete because of the detailed written feedback that is given automatically by OU tutors to their students. {style} {style:type=span}2.{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}An ethnographic study was also undertaken by the Educational Technologist and the Programmer i.e. they both became History students and answered the questions. This part of the project revealed that maximum marks can be gained only when the student understands “the rules of the game”. Once the researchers had undertaken a knowledge elicitation session with the History tutor, they were able to increase their grades from a C to an A because they were aware of these “rules of the game”.{style} {style:type=span}3.{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}Incorporating feedback about “the rules of the game” we felt should be included in the feedback, perhaps with a separate button that could be turned on or off by the student. Surprisingly this idea is being resisted, not by OU tutors, but by others from a number of universities where the system has been demonstrated.{style} {style:type=span}4.{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}Multiple disciplinary teams are needed to conduct this type of work.{style} {style:type=span}5.{style}{style:type=span|font-family='Times New Roman'} {style}{style:type=span|font-size=8pt|font-family='Arial','sans-serif'}Editing tool that tutors can use directly is an essential {style}{style:type=span|font-size=8px}component of this system.{style}{style:type=span|font-size=12px|color=#000000|font-family=Times New Roman} {style} 6. {style:type=span|font-size=8px} A pedagogical model of feedback has been developed which is now open to test. We are waiting on findings from student usage.{style} | |
| Object number 0 of type XWiki.XWikiRights added | ||
| users | ,XWiki.DeniseWhitelock | |
| allow | Allow | |
| levels | edit | |
| Object number 1 of type XWiki.XWikiRights added | ||
| allow | Allow | |
| levels | view,comment,edit,delete | |
| groups | XWiki.XWikiAdminGroup | |
| Object number 2 of type XWiki.XWikiRights added | ||
| allow | Allow | |
| levels | view,comment | |
| groups | XWiki.ProjectMembers | |
| Object number 3 of type XWiki.XWikiRights added | ||
| allow | Allow | |
| levels | comment | |
| groups | XWiki.ExternalPartners | |
| Object number 4 of type XWiki.XWikiRights added | ||
| allow | Allow | |
| levels | view,comment | |
| groups | XWiki.AdvisoryBoard | |
| Object number 5 of type XWiki.XWikiRights added | ||
| allow | Allow | |
| levels | view,comment | |
| groups | XWiki.UserGroup | |
| Object number 6 of type XWiki.XWikiRights added | ||
| users | XWiki.XWikiGuest | |
| allow | Allow | |
| levels | view | |
| Object number 7 of type XWiki.XWikiRights added | ||
| allow | Allow | |
| levels | view,comment | |
| groups | XWiki.XWikiAllGroup | |
