Case Story: Open Mentor

Summary	Open Mentor
Group / workshop	Formative e-Assessment	Status	seed
Project details...

Situation

What was the setting in which this case study occurred?

Open Mentor was built to assist tutors at the Open University to provide constructive feedback to students in order to help them improve their work but also to give socio-emotive feedback as well.  This is because praise is important to distance learning students who do not have full knowledge of their peer's progress and to inform tutors that the mark does not speak for itself.  The precursor to Open Mentor was e-Mentor which provided a proof of concept system which was internally funded by the OU. It stripped out all the tutor's comments made on an assignment and classified them into four Bales' categories (see details of Bales in Task Section). The important finding here was that the system worked for any type of assignment, whether it be Mathematics, Science, from the Arts Faculty or Education. 

Open Mentor was then designed to give feedback to the users in a graphical form and had an interface where assignments could be submitted.  The system was intended for tutors to use, not only at the Open University, but in any establishment and can be customised to suit any HE marking system (grades or numbers etc.).  The students' essays have been assessed from  the OU, Robert Gordon University and Manchester Metropolitan.  

Any assignment can be uploaded to this web system if it is in a web format. The advantage here is that the course does not have to be delivered via the web. I believe the use of Open Mentor is now more timely as most HEs are asking their students to deliver their assignments electronically. 

Task

What was the problem to be solved, or the intended effect?

Open Mentor was designed to provide tutors in the OU and RGU with a tool that would assist them with assessing the support and guidance they were giving to students (i.e. their written comments) on their assignments with respect to the mark awarded for that assignment.

The feedback to tutors would be in a graphical form illustrating how many comments they had provided to the student which were grouped into the four major Bales' categories.  This analysis would be compared to an ideal set of comments which would be given to a student for the particular mark awarded (see screen dump).

The rationale to provide this type of feedback to tutors in order to help them to support the feedback to their students, is given below. the important point to stress here is that tutors know their subjects and are competent at awarding grades and we did not want them to feel threatened in this respect. However, we wanted to help them provide more constructive feedback to assist the students improve in their next assignements. We wanted students to start to understand what they did not know and to enter a dialogue with their tutors.

Students need a balanced combination of socio-emotive and cognitive support in their feedback from teaching staff, and the feedback needs to be relevant to the assigned grade. In order to provide feedback, Open Mentor has to analyse the tutor comments.  The classification system used in Open Mentor is based on that of Bales (1970). Bales’s system was originally devised to study social interaction, especially in collaborating teams; its strength is that it brings out the socio-emotive aspects of dialogue as well as the domain level. In previous work (Whitelock et al., 2004) we found that the distribution of comments within these categories correlates very closely with the grade assigned.  Bales’ model provides four main categories of interaction: positive reactions, negative reactions, questions, and answers. These interactional categories illustrate the balance of socio-emotional comments that support the student. We found (Whitelock et al., 2004) that tutors use different types of questions in different ways, both to stimulate reflection, and to point out, in a supportive way, that there are problems with parts of an essay. These results showed that about half of Bales’s interaction categories strongly correlated with grade of assessment in different ways, while others were rarely used in feedback to learners. This evidence of systematic connections between different types of tutor comments and level of attainment in assessment was the platform for the current work. The advantage of the Bales model is that the classes used are domain-independent – we used this model to classify feedback in a range of different academic disciplines, and it has proven successful in all of them. An automatic classification system, therefore, can be used in all fields, without needing a new set of example comments and training for each different discipline.  A second point is that Bales draws out a wider context: we found that as we started to write tools that supported feedback, we began to question the notion of feedback itself. Instead, the concept seemed to divide naturally into two different aspects: learning support and learning guidance. Support encourages and motivates the learner, guidance shows them ways of dealing with particular problems.

Actions

What was done to fulfil the task?

1. Understanding the stakeholders needs In order to build the first storyboards for Open Mentor and to ensure the software would meet the needs of both tutors and students, we devised two questionnaires, one for tutors and the other for students.  44 tutors from Kings College London, Manchester Metropolitan, The Open University and Robert Gordon University completed the tutor questionnaire while 47 students from The Open University and Robert Gordon University responded to a questionnaire which was designed to understand how students reacted to tutor feedback.
2. Stakeholders’ needs identified Both tutors and students agreed that comments should reflect the grade awarded, which is a basic premise of the Open Mentor system.
   
3. System architecture The resulting Open Mentor architecture is based on the following main components:

• A data source for course information and lists of students and tutors
• A data source for use within Open Mentor, to store assignments, submissions and classified comments
• A classifier which can categorise tutor comments
• An extractor which can read tutor comments from word processed files
• An evaluation scheme description which defines the classes of comments, the grading bands and the expected benchmarks
• A logic component which applies the evaluation scheme to the classified

comments

The advantage of this is that different institutions can write their own components and add them into the system without having to do any modification of existing code – this reduces the risk of errors and other problems.

Results

What happened? Was is a success? What contributed to the outcomes?

Open Mentor is implemented using Java, and runs as a web application, enabling people to use it in any location.   Again, open source influenced the direction of the project; initially we had used open source as a kind of library of components that we could re-use. Later, particularly when we moved to Spring, we found our system became much smaller, as we could plug our developments more easily into larger frameworks. We also moved to a point where we can contribute to open source: our developments of the Apache POI-based code for extracting text from Word files exceeded the capabilities of the standard distribution. UK higher education has an important new dissemination route for its developments in these channels – however, the same resourcing issues that led to this situation still need to be addressed.

Lessons Learned

What did you learn from the experience?

Analysis of stakeholders’ needs and user testing of the interface and system is essential throughout the design process. This JISC funded project was highly recommended for its user testing and participant design methodology. Open Mentor has been used with training sessions for tutors. The biggest surprise the tutors encountered how they needed to use more praise in their comments to students. Many believed that the mark spoke for itself neglecting the reassurance that students need as they do not necessarily know how their mark relates to the rest of their group’s progress.

Licensing

This work is licenced under a Creative Commons Licence.