Case Story: Developing Formative Assessment for H812: Postgraduate Certificate in Academic Practice

Summary
Group / workshop	Formative e-Assessment	Status	seed
Project details...

Situation

What was the setting in which this case study occurred?

One of the strands of the Computer Assisted Formative Assessment (CAFA) project led by Denise Whitelock at The Open University was to work with Moodle to produce a set of formative assessments. This study forms part of that strand. The course that volunteered to participate in this part of the project was H812 Postgraduate Certificate in Academic Practice (PCAP).  Its first presentation is in October 2006 and incorporates a number of interesting features in the bullet points below:

• Benchmarked against HE Academy professional standards
• Designed around learning objects
• Designed for easy up-dating, reversioning and customisation
• Designed to afford the introduction (2007 onwards) of several APEL routes
• Designed to engage any members of OU staff who support student learning with pedagogic and curriculum issues
• Designed with the advice of a large group of OU and external stakeholders
• Highly customisable, in that for any given theme, course members do web searches to identify materials relevant to their situation
• Intention to provide computer-assisted formative assessment
• Moodle-based
• Podcasts
• Process intensive design (no course reader or other print material)
• Provision of a small-scale enquiry block
• Strong emphasis on e-learning The study set out to explore the Moodle quiz facility for producing formative assessments within a relatively rapid production schedule.

Task

What was the problem to be solved, or the intended effect?

Aims

The aim of this project was to produce a working set of e-assessments for H812 to run within Moodle for the first presentation of the course in October 2006.

Objectives

The key objectives of this project were as follows:

• To discuss with the course team how to introduce the elements of Computer Assisted Formative Assessment into the production of H812.
• To design a set of e-assessments with feedback for Blocks 1 and 3 of the course
• To implement the e-assessment design within Moodle which was the platform of choice for H812

Actions

What was done to fulfil the task?

An agile methodological approach was adopted rather than a plan driven methodology since the former supports adaptation rather than prediction.  This consideration was crucial since the H812 course was piloting the use of Moodle for e-assessment and stretching it as far as it would go to achieve the desired pedagogical aims.  Agile methods are less risky than a plan driven approach where it can be hard to adapt to necessary changes in project emphasis as the programme of work progresses. The agile process consists of five main phases as shown in Figure 1 below.  It is an approach that has been adopted by myself in a piecemeal fashion over time in partnership with Robert Gordon University in the Open Comment and OpenMentor Projects.  
 Figure 1: Adapted agile process

Implementation

The design and implementation consisted of:

• A set of interactive questions with feedback for Blocks 1 and 3
• Two distinct question types were designed. These being multiple choice and fill in the missing words
• Moodle quiz with the added feature of Moodle’s ‘cloze’ type question was used to implement these two question types

Proof of concept was therefore established and the implementation met the project’s requirements.  It was then passed on to LTS for roll out to students.   LTS was not able to support the missing word question type with drag and drop functionality.  This meant that the questions then had to be rewritten by the course author as simple multiple choice questions.

Results

What happened? Was is a success? What contributed to the outcomes?

The course team decided that the CAFA tasks would be made compulsory and contribute to 20% of the eTMA marks.  Findings from other strands of the CAFA Project, such as the work undertaken with T180 and AA308, confirmed that a disappointing number of students completed CAFA tasks when there was no compulsion.  Hence a clear decision was made to introduce CAFA tasks that would be completed by all students and assist with their mastery of the subject.   At first sight there appears to be a contradiction here between formative assessment which should not contribute to the final student mark and a regime that insists on an element of summative assessment.  In my opinion, a good compromise was arrived at to preserve a formative and mastery learning intentions of the course team. The strategy they adopted was as follows:

• All assessment tasks were made compulsory since they contribute to 20% of the eTMA marks
• Feedback was provided on all attempts for each task.  In this way, students would know how they were progressing and also obtain information about why some of their choices were rejected by the CAFA programme
• All students could attempt the CAFA tasks as often as necessary
• A 100% success rate on the CAFA tasks was needed before the eTMA requirements could be met

The multiple choice questions were implemented within Moodle and are available for students in the first presentation October 2007.    Multiple choice questions are handled reasonably well by Moodle.  However there are a number of limitations. These include problems with randomisation of questions.  Moodle can only randomise choices for the whole quiz and not for individual questions. It is difficult to use formatted text in the feedback for each choice within a question and if the feedback is the same for different choices, it has to be entered repeatedly. This is a maintenance problem which should be addressed by the Moodle development team. The developer also discovered that using a single question per page results in an unnecessary extra ‘submit page’ button. This could easily be eliminated with a change to the Moodle system code. Fill in the missing type questions are ideally implemented using a drag and drop style interface.  This is not supported by Moodle.  One of the reasons for the absence of drag and drop within the Moodle system could be a reliance on javascript/DHTML code which makes cross browser support more difficult.  However, our software developer implemented this question type using Moodle’s ‘cloze’ type question. This function facilitates the embedding of drop down lists of choices within a body of text.   There are limitations to this ‘cloze’ style entry of questions which maybe difficult for some developers to use. The feedback for embedded choices may also appear behind other drop down lists.  This is a DHTML code problem and deserves further consideration.  Unfortunately LTS considered this question type to be unstable and was not supported on the OU versions of Moodle.

Lessons Learned

What did you learn from the experience?

Outcomes

The following major outcomes are noted:

1. The two types of questions produced in Moodle for the H812 course team were favourably received.
2. Unfortunately only multiple choice questions could be supported by LTS and are the only type available to students in the first presentation of the course.
3. The course team have been made more aware of both the potential and limitations of the Moodle quiz facility.
4. The agile methodological approach facilitated the adaptation of the Moodle quiz facility.

Conclusions

The following conclusions are noted:

1. Using Moodle for the implementation of interactive questions does provide developers with a coherent system to create, deploy, manage and maintain questions. It also provides good features for retrieval of student responses and scores. Therefore for the deployment of very simple quizzes, Moodle is a good tool.
2. The use of more advanced questions in Moodle requires significant development.
3. The limitations of the Moodle quiz will have a restricted appeal for course developers.
4. The pedagogical driver to introduce meaningful and exacting formative assessment has been restricted with the introduction of Moodle as the VLE.

Licensing

This work is licenced under a Creative Commons Licence.