Case Story: String comparison in language learning

Summary	Using string comparison for error detection in language learning
Group / workshop	Formative e-Assessment	Status	seed
Project details...

Situation

What was the setting in which this case study occurred?

Teaching Spanish as a wild-card module for undergraduates studying other subjects at a UK university. The module grade counts towards the final degree award. This case focuses on the need for students to practice written language independently and receive feedback on errors in order to improve their language skills

Task

What was the problem to be solved, or the intended effect?

The students answer randomly-generated, translation-based questions, grouped into exercises which focus on specific areas of grammar. There are very large numbers of students taking the Spanish module making it exceedingly time-consuming for tutors to provide detailed individual feedback. Standard parser-based solutions were not feasible since these tend not to be able to cope in the face of poor answers.

Actions

What was done to fulfil the task?

A bespoke string (sequence) comparator was designed. Rather than using parsing the system uses fine-granularity sequence comparison to compare correct language strings to a user’s answer. With such a technique generic - but detailed - feedback is always given, no matter how confused the user's answer is.

Students answer questions and the comparator marks up errors in their input using colour coding (and font style) to highlight the different types of error: incorrect words, misspelt or misconjugated words, omitted words, redundant words and incorrect word-order.

If an answer contains errors the student is given a second attempt in which to correct the submission based on the feedback received. The sequence comparator is language-independent and feedback is therefore generic in nature (i.e. no specific grammatical clues are given), but this was considered preferable to using a system which would not be able to handle muddled input.

Results

What happened? Was is a success? What contributed to the outcomes?

Despite the lack of grammatical information in the feedback the system works very well. There is virtually always an improvement between students’ first and second answer attempts, indicating that the unusual feedback method is effective; but there is also measurable improvement over the course of an exercise.

For data analysis all student data files for every exercise submitted were divided into a number of separate sections and the answers given in each of the sections were examined. It is apparent that as students progress through an exercise their answers become more accurate while their thinking time decreases. This is a definite indication of improvement in language learning capacity.

The students themselves evidently found the system useful as evidenced by the extra effort they put in. The system sets a minimum and maximum number of questions to be attempted for each exercise. Students can stop after completing the minimum number of questions, but can carry on (up to the maximum) if they wish. On average students attempted 50% more questions than they were required to do.

Lessons Learned

What did you learn from the experience?

Previously students had to translate batches of sentences into Spanish, and would receive their marked work back some 2 weeks later. If grammar points had been misunderstood they were likely to repeat their errors throughout the exercise, thus contributing to their misconceptions becoming ‘fossilised’. Furthermore, given the time lag, there was no guarantee that students would look at feedback properly once corrected versions were returned. It seems there is considerable potential here to develop this approach. It is bespoke, so not quickly transferable, but will work with a variety of exercise types for a large number of foreign languages. Has application to pre and post-16 contexts and is currently being trialed in 2 schools.

Licensing

This work is licenced under a Creative Commons Licence.