Case Story: ALPS School of Medicine

Summary
Group / workshop	Formative e-Assessment	Status	seed
Project details...

Situation

What was the setting in which this case study occurred?

Fifteen medical students failed exam. Put into clinical setting (ward) and immersed in learning and assessment about 8 weeks prior to a retake of the same assessment format (find a case and report decisions made about patient to an assessor). The skills the student needs to work on are identified through a formative assessment regime (these areas may not be apparent to the student). Students are then in a better position to pass the exam.

Task

What was the problem to be solved, or the intended effect?

An assessment in this case – the student identifies the situation they wish to be assessed in eg a GP surgery where a patient shows up with abdominal pain, student decides this is good opportunity to demonstrate an abdominal examination. Assuming they get permissions they are observed and assessed with scores (1-9) and comments (formative feedback) applied according to a set of criteria. These can be undertaken verbally or by typing. Sometimes this is not undertaken and the evidence is lost. During the assessment the assessor has the PDA (this is usually owned by the student). The PDA is not locked down but runs a specific piece of software called MFORM. This allows the delivery of assessments remotely with results sent back to a server for recording and processing. Twenty assessments was found to be too many. On average the student would do fifteen assessments so two or three a week.

There are three ‘forms’ each covering a clinical methodology. Clinical skills (poking students), clinical reasoning (working out what is wrong with them, triage processes to arrive at a diagnosis) and communication.

Mixed student feedback about this at beginning of placement (over assessment, overload, welcomed feedback opportunities, technical phobia). Ten of the fifteen passed the exam second time around.

A web site allows the students to review their feedback and scores at any time. This is not enforced.

This triggers a feedback process. Students track down the assessors to discuss their scores and comments or to set up a further assessment to address weakness in specific areas.

It was compulsory to undertake the process, it was not compulsory to interact with the feedback and scores, the scores were not summative.

Culturally one problem is that formative assessment in the medical domain is not commonplace.

Some students started the process early and were quick to address their weak areas. Some students were lax and left just 4 weeks to undertake the entire process. Students realised this was a significant workload.

No definite patterns for conclusions as the sample size was too small.

Post event students were questionnaired about their experiences. The PDA ownership novelty wore off quickly. The PDA keyboards were difficult to use. Audio should have been taught as the preferred method. The wider PDA functionality was used.

Discarding the PDA element the assessors and student felt that the structured formalised assessment approach was very helpful.

Calibration of the assessors use of 1-9 scoring was not formally enforced in terms of equality between assessors however medics are used to using such sliding scales.

The situation is highly stressful as failure a second time around means the end of 5 years of education and no practising license.

There is a lack of personalised learning opportunity in medical education. The approach addressed this from both students and assessors.

Assessors reported that the process reflected on their teaching practice.

The process is being refined and repeated and conclusions could be drawn at a later date regarding effect on pass rate. At this stage it appears to make a difference.

As this is a 10 week process there is much more going on than the assessments. If a student has identified a weakness through an assessment there are multiple opportunities to take remedial action (not just through taking another assessment).

Formative aspect is reflection required by student to identify their weaker areas and work on them.

There is a good deal of research about how to do this sort of thing and a PDA probably limits this but the fact that this process produces a pile of evidence for the students to look at and do something proactively.

Actions

What was done to fulfil the task?

Results

What happened? Was is a success? What contributed to the outcomes?

Lessons Learned

What did you learn from the experience?

Licensing

This work is licenced under a Creative Commons Licence.