Pattern: Try Once Refine Once
Pattern: Try Once Refine Once
| Summary |
A two-step question-answering system which encourages students
to consider their initial answers to skills-based questions very
carefully, and, on receiving feedback on their errors, to give as much
thought to the refinement process. |
||||
|---|---|---|---|---|---|
Status 
| alpha | Confidence
| 2 | ||
| details... | Group | Formative e-Assessment
|
|||
Problem
Large numbers of students on a skills-based course. Lack of immediate feedback for students leads to fossilisation of errors and misconceptions - however providing immediate feedback in an iterative fashion can also hinder effective learning since students are able to "grope their way" step-by-step to a correct solution without necessarily having to think about each answer as a whole.Context
Skills-based learning situations where multiple misconceptions in exercise answers are possible. Particularly applicable to foreign language learning, but should also work for other skills-based fields. The range of assessment types this approach might be suitable for would be those in which student answers can contain multiple errors, for which detailed feedback indicating the source and type of each of the errors can be generated/given, without revealing exactly what must be done to correct them.Solution
Students are posed questions of a type which elicit answers that can contain multiple errors. If a student's answer is entirely correct a mark of 100% is awarded. If their answer contains errors, a mark is given which contributes to a percentage of the total mark for the question, along with detailed - yet generic- feedback on the location and type of the errors. Students are then permitted a second attempt in which to refine their answer. The mark for the 2nd attempt contributes to remaining percentage of the total mark for the question. Feedback on any remaining errors is also given, along with the correct answer(s). No further attempts are permitted.
The two-attempt limit and unequal weighting of the marks for the initial attempt and the refined answer are crucial to this pattern, since they prevent students from adopting a mindless iterative approach, in which they begin with a "stab-in-the-dark", then allow the system/tutor to guide them step-by-step to the correct answer (often via numerous minimally-altered attempts).
The marks ratio can vary, but showing a distinct favouring for the first attempt works best - ensuring that students give careful consideration to all components of their first answer, and equally careful consideration to improving it in the face of the diagnostic feedback. If the ratio is skewed too far in favour of the second attempt then students tend to exhibit less care over the construction of their initial answer. If the ratio is skewed too far in favour of the first attempt then students are less inclined to try and correct non-perfect answers.
The marks ratio could be adjusted according to the amount of information in the feedback. The less information in the feedback the higher the second mark should be, the more information in the feedback the less the second mark should be.
(Sequence diagram from http://www.websequencediagrams.com
Related Patterns
list other patterns related to this one, under categories such as component, assisting, conflicting, uses this, etc.
Support
* Source and Additional Supporting Cases
Source Case (chosen from Case Studies)
Other Cases (chosen from Case Studies)
Links to External Case Stories & Examples
* Rationale (theoretical justification)
Theoretical justification.
In terms of the Conversational Framework (Laurillard 2002) we see here the learner's action to achieve task goals, feedback on action, and reflection on feedback leading to a change in the learner's conception. The Conversational Framework provides no specific justification for the division of marks or form of feedback.In terms of Black and Wiliam's (2008) theory of formative feedback, this is an example of key strategy 5. activating students as the owners of their own learning.
A particular clue as to why the assessment regime proposed in this pattern might work is provided in Hattie and Timperley 2007 who write: “The degree of confidence that students have in the correctness of responses can affect receptivity to and seeking of feedback. Kulhavy and Stock (1989) noted that if confidence or response certainty is high and the response turns out to be a correct one, little attention is paid to the feedback. Feedback has its greatest effect when a learner expects a response to be correct and it turns out to be wrong. As Kulhavy and Stock noted, “high confidence errors are the point at which feedback should play its greatest corrective role, simply because the person studies the item longer in an attempt to correct the misconception” (p. 225).” Because 75% of the marks will be given for the first attempt the students are likely to give answers in which they have a considerable degree of confidence – so, if the answer is then found to be incorrect, then this is a situation where the feedback will be most effective. This pattern contains several features which meet Nicol and Macfarlane's (2006) 'principles' of good feedback as core to the 'source of formative assessment', which enables learners to 'make evaluative judgements about their own work' (Nichol, 2008). In the argument made by Nichol and Macfarlane, learner self-regulation is fundamental within formative processes. This pattern meets the following 'principles' by which learner self-regulation is achieved (not all 7 principles will be met equally in all contexts - these are the 5 significant ones here):
- facilitates the development of self-assessment and reflection
- delivers high quality info to students about their learning
- encourages positive motivational beliefs and self-esteem
- provides opportunities to close the gap between current and desired performance
* Verification (Solutions that were derived from this pattern)
Scenarios / solutions which were developed using this pattern.
Notes, Links and References
Liabilities, potential risks, extensions, expected side-effects
In the CALL exercises from which this pattern was drawn, the ratio of marks between the first and second answer attempts was 3:1. This proved optimal for the original situation but is obviously easily altered for other assessment types. The "try once, refine once" approach led not only to marked improvements between first and second answer-attempts, but more importantly to demonstrable improvement in accuracy (and speed) of answering as users progressed through exercises. In other words, students became able to formulate their foreign language sentences more accurately and with greater rapidity, which is a good measure ofsuccess in language learning.
It should be noted that the CALL questions (English sentences to translate) were generated randomly and students could do each exercise in a single sitting or in multiple sittings over the course of several weeks. Thus it was not the case that improvements were down to question-ordering or the effects of short-term memory. Furthermore sentence-types could be fairly complex, and students had to attempt to get all aspects of a sentence correct, so it was not simply a matter of concentrating on a single grammatical aspect such as verb endings.
It is also worth mentioning that students often chose to do far more
than the minimum number of questions per exercise than they were obliged to do, because they found the system helpful and were aware that they were improving by using it.
Licensing
This work is licenced under a Creative Commons Licence.
