Case Story: AA308 Case Study: Experiences of a course team producing formative e-assessment for the first time

Summary
Group / workshop	Formative e-Assessment	Status	seed
Project details...

Situation

What was the setting in which this case study occurred?

The case study reported here was based in the Arts faculty. Key stakeholders who were involved in producing and sampling  a formative e-assessment task for AA208 course were interviewed.  The main driver for this particular development came from the faculty itself which had already identified the need for a substantial development in the use of formative interactive assessment. The following advantages of using online assessment as a means of delivering formative assessment were cited:

• a change from the traditional SAQs and provides a more varied learning experience for the students
• more likely to be fun to use and to enhance the learning experience
• helpful in increasing student motivation and, ultimately, retention.
• suitable for a range of abilities.
• can be used flexibly, both as consolidation material and as a revision tool for exams.
• allows for more interactive learning and encourages more active student participation
• more students now are attuned to an online learning environment and expect the OU  to keep abreast of  technological development

Against this general background within the Faculty, a software developer in LTS had a program that could be used for online questions and, in discussion with members of the Arts faculty, established that the then-forthcoming Philosophy course would be a good place to use it. The program had already been successfully used in a FELS course, so it had been piloted. There were also enough members of the AA308 course team sufficiently enthusiastic about the idea to drive this initiative forward: “We were asked to try it out and we were convinced it was worth having a go”.

Task

What was the problem to be solved, or the intended effect?

Aims

The aim of this CAFA funded project was to extend the understanding of what it means to producers of an e-assessment when it is being developed for the first time within a Faculty.  To then document the generic lessons that can be learned from this endeavour and  feed them back to the University.

Objectives

• To interview key stakeholders involved in the production of an online formative assessment for the first time in the Arts Faculty
• To identify both the enabling factors and barriers associated with e-assessment production

Actions

What was done to fulfil the task?

A case study methodology (Gomm et al, 2000) was adopted in order to create a framework within which to assess the barriers and facilitators of e-assessment practice.  The method adopted fell into the following three phases which are documented below: ·          Preparation phase·          Collection phase·          Analysing phase Preparation PhaseThe key stakeholders were identified as: ·          The Associate Dean, Teaching & Learning, Arts Faculty·          Academic members of the course team·          The Director of Innovation, Arts Faculty·          The course manager·          The software designer·          An Associate Lecturer·          A student  The Interview questionnaire was constructed from five basic research questions which would probe the “who”, “what”, “where”, “how” and “why” for each e-assessment application.

The “Who” Questions explored the personal aspects of e-assessment development.  These probed for example the roles and responsibilities assigned to all the participants.  The “who” questions were strongly linked with the different stakeholders to find out what the role and responsibility of the individuals were.

“What” Questions were asked in order to ascertain more about the nature and type of e-assessment which was being delivered .

The “Where” Questions addressed the delivery issues and the “Why” Questions were of an explanatory nature and explored the motivation behind any particular e-assessment development and evaluation. Collection PhaseThe collection phase started with visits to OU personnel on campus and then linking by telephone to ALs and a student.  The interview protocol was refined after piloting.  Semi structured interviews were conducted while using the data collection protocols for each of the stakeholders. Questions were adapted and added when the interviewer felt that the data collection protocol was not sufficient for a particular interview. All interviews were recorded by an audio tape or a digital recorder and then transcribed.   Analysing PhaseThe final phase of the project analysed the transcripts and a summary of findings produced.

Results

What happened? Was is a success? What contributed to the outcomes?

Enablers

In the university as a whole, the development of the VLE, and IT developments more generally, were seen as providing an appropriate climate in which to develop this sort of extension to the traditional type of teaching materials. There is also now a greater awareness of the need of the OU to maintain an up to date profile with regard to the IT skills that are increasingly being recognised as an essential element of an undergraduate degree. “I think that the message from students is that we are a university that provides degree level and degree qualifications and it is no longer the case that the degree can be attained without some kind of ICT access. We’re now almost ten years since the Dearing report recommended that all students have a computer of their own”. “ [The current model of distance tuition] has served us well, but it runs the risk of being dated and if we can do more with the technology to enhance the learning experience, then we should do this as a university”.

Within the Arts faculty, the development of more flexible, online materials also fits well within their general ethos: “We’ve always said in this faculty that we believe in media-rich teaching. Media rich teaching involves an appreciation of what learning can be done in various different contexts and different media”. “I think the use of synchronous conferencing and technology for the development and assessment of students’ oral presentation skills and their skills of working together is something we are very interested in”.  “Although the AA308 tool is limited…it certainly feels very different. It offers people more opportunity to play and think why does this come under this category and not that category?” There is also a sense of students’ expectations changing and of wishing to provide an enhanced learning experience for them. Within the Philosophy department, there was the combination of appropriate software being available and a course team willing to experiment with it. An important enabler was felt to be that not only were there people who were able and willing to engage with new technology, but also that there was a leadership prepared to support it.

Barriers 

The attitudes of some OU staff and students can act as a barrier to the acceptance of new technologies.  “The problems and obstacles for me are in terms of actually getting central and academic staff to realise that the teaching environment is changing…there is some resistance from a minority of the faculty to the concept of ICT and new technology” There are other access issues still to be sorted out – such as whether students will need to have access to broadband rather than dial up in order to access some material. There are also issues of ongoing staff development: “…once one gets away from culture change, there are enormous problems of staff development, there is a problem with certain sections of the Arts AL body of resistance to new technology as seen in recent correspondence to Sesame”. Another barrier could be that low expectations of what might be achieved through this mode of assessment, could hinder the urge to experiment with it: However, a potential barrier is whether the technology will be able to develop fast enough to meet the expanded requirements of it. Whilst the formative assessment tool in AA308 was deliberately limited to revision of basic points, there will eventually be a need to develop more sophisticated skills.

Perceptions of AA308 students 

Three students were interviewed for this case study (as distinct from those who were surveyed by IET in 2005). Two of these worked in ICT; of these two, one only noticed the assessment exercises, towards the end of the course, when he was revising: “One of the problems with courses is that there are often a lot of materials, books, forums etc, so it’s trying to understand which bit you should focus on because you can never do them all, usually. But I eventually did come to this, particularly when I was revising and I wished I’d used it more, it was very effective and I only really used it at the end part of the course”. The other did not really notice them at all, though when I sent him a sample of the exercises, he said he would have used them if he had realised what they were like: “Certainly the format I’ve seen today was fine…I would be happy using it and I would probably want to on these courses… I have a bias to using new technology anyway”  The third student said that there was such a volume of course material that she missed out on anything that was optional in order to save time. However, having subsequently taken a course in another discipline which required much more online material, she had become convinced of its benefits!Although the existence of the formative assessment material is flagged on the e-desktop, it seems that it is not sufficiently salient to be picked up by the students: Once the students were aware of what the exercises were like, they were enthusiastic about them, so a more active promotion of these elements of the course should help to address this issue.

Perceptions of an AA308 tutor 

The tutor interviewed was aware of the e-assessments.  He felt they were pedagogically sound but did not promote them to his students. He highlighted the dilemma of the university in seeking to move forward in using new technologies to enhance the quality of student learning where some sections of the student population are somewhat technophobe: “I think that if you don’t have as many takers as you would expect, I think it’s the technology that holds them back. People don’t access it because they don’t use their computers much or perhaps with some students, they don’t even have one”.

Lessons Learned

What did you learn from the experience?

A facilitative climate within the Faculty assisted with this formative  e-assessment development.  The main drivers for adoption included:

• Preparation for the introduction of a VLE
• Developing more flexible online materials to add to their suite of teaching resources
• Students’ demand for more e-learning material
• Creative pedagogy in the Faculty
• Staff willing to experiment with teaching resources and exhibit leadership in this area

The tension between pedagogical push and technological drag is evident.  The tool used here will not be supported in the future although more advanced than the Moodle quiz it does not meet the demands of the teachers. However, what is encouraging is the academic author’s enthusiasm and expertise when writing the questions.  The author stated “writing the question can be positive.  Seeing the subject from different angles (question formats) makes you think.  Thinking of interesting questions.”

The barriers to a disappointing rate of adoption included:

• AL attitudes to technology
• Student attitudes to technology
• Uncertainty about the software tools to be supported in the future by LTS
• Low expectations of e-assessment for Arts courses
• Staff development

The development was a time consuming process and took longer than expected. It is disappointing to find a low uptake of this expensive learning resource by the students.

Licensing

This work is licenced under a Creative Commons Licence.