Driving change in assessment and feedback

Kim Catcheside

Assessment is core to the learning experience: how students are assessed shapes their understanding of the curriculum and determines their ability to progress.

The National Student Survey scores for assessment and feedback are consistently low and are one of the drivers for Jisc’s Assessment and Feedback programme which looks to enhance this vital component in learning through technology.

Gill Ferrell

It’s a thorny problem for institutions to tackle because it is so fundamental to the learning experience. It’s such an important part of what they do yet it’s quite difficult to get a grip on because assessment and feedback is an area where the policy making is actually quite devolved within an institution. At the heart of it I think it’s really about good educational design.

Kim Catcheside

For teaching staff though, valuable assessment and feedback takes a significant amount of time. Given the increasing student numbers per staff member, smaller budgets and greater learner expectations, institutions delivering higher education are under pressure to find better methods, without increasing the student’s workload. What role can digital technologies play?

(Music)

Kim Catcheside

Hello and welcome to Jisc On Air, I’m Kim Catcheside and in this edition, I’ll be exploring the issues with traditional feedback and assessment practices and consider some early stage Jisc-funded projects which draw on technologies  to improve the students understanding of the curriculum and their progress through? it. As we’ll hear, using technology for assessment and feedback in higher education is proving vital in enhancing transferable skills such as self-evaluation, critical thinking and collaboration – skills that are essential for today’s workplace.

(End music)

Kim Catcheside

David Nicol, Emiritus Professor at Strathclyde and Visiting Professor at the University of Ulster carried out extensive research as part of REAP, which stands for re-engineering assessment practice. His research into assessment and feedback produced influential findings that have fed into current understanding..and the underlying aim of feedback

David Nicol
We need to see feedback, not just as a process that’s intended to improve learning that there’s a bigger idea behind feedback, in that it should enable students to develop the capacity over time to make judgements and evaluate their own learning and to produce quality work themselves without the help of the teacher. My original work started with seven principles

Kim Catcheside

I realise I’m asking you to do a difficult thing, but if you had to do a desert island discs, which would you choose?

David Nicol

Well, I think the underpinning idea behind all of the principles is that we need to think about the way that students actively engage with the feedback information that teachers provide. The normal idea of feedback is to deliver some written comments to students – it’s not enough – we must think how students construct meaning from those comments and how they can act on them – and we would hope that students can make judgements about their own work because when they go into professional practice, students set their own goals, evaluate their own progress, and we need to ask are we developing this in higher education?

Kim Catcheside

The employer is increasingly playing an important role in the design of all good feedback and assessment. At the University of Exeter. Michele Shoebridge is the Deputy Registrar.

Michele Shoebridge

We know that academic excellence is important but you know employers are looking for all sorts of other things, they’re looking for students that can demonstrate that they can solve problems, that they can work in teams, you know that they can identity key trends, or they can do some research on their own. So there’s lots and lots of things that go beyond I think the classification of your degree and if Exeter can get that reputation with employers that will be really good for us.

Kim Catcheside

At Cornwall College, Tony Harris, Project Manager of FAST, which stands for feedback and Assessment Strategy Testing, and is supported by Jisc, is equally motivated to develop skills for the workplace through the assessment process.

Tony Harris

I think what’s driving this, is employers. Employers for a long time and more recently have been saying that they are not convinced that what we are doing is in tune with what employer need and demand is.  We are very much an employer led organisation, we are responding to the need of employers. The other side of this is the growing use of technologies. I mean we are seeing it all over and we haven’t harnessed it properly. We know students use different technologies and teachers use different technologies but there is no way that we really thought out how we use technology in a structured way in teaching and particularly in assessment and feedback and they’re the two key issues. And what we want to do is one, respond to employers and two, try to get a way forward for the way that Cornwall College actually harnesses the new technology

Kim Catcheside

Tony and his team are tackling the issue head on, they’re pushing students to use technology and new platforms to deliver their work, and in doing so, encouraging them to develop important skills for the workplace.

 

Tony Harris

The reasons we embarked on this project was because we found that over 90 % of overall the assignment submissions were submitted via Microsoft word. What we’ve done is, we’ve removed that, the option to submit in the word format. So what we’re actually saying that all formative and summative assessments would now be delivered using various forms of technology some of which are imposed, some of which are self-selected by students.

Kim Catcheside

Tony’s colleague Adele Oakes is Programme Manager of FAST and Access Coordinator at Cornwall College.

Adele Oakes

One of the assessments for our module, which is a reflection on a group project they take part in, instead of it being a written piece of work that they hand in, they as a team will be asked to do a constant reflective blog throughout their team challenge. Reflecting on things as they happen so they’ve got something to actually go back and look at. And actually they’ll be doing their reflection as video diary and talking it through that way rather than giving me a written reflection because actually what I’m looking for is the transferrable skills which I’ll see more on screen than I will on paper for a lot of the skills that I’m looking at for that.

Tony Harris

One of our fundamental principles is that we believe that if you’re going to assess using an essay is then what you’ll get is probably good essay writers and is that what employers are wanting these days? The answer is coming over loud and clear no that’s not necessarily what they’re wanting. So if you’re going to submit a work using different technologies that’s more in line with employer expectations

Kim Catcheside

One of the issues that Higher education institutions face with the increasing awareness that good assessment can also be good for learning… is bringing academic staff with them. At the Institute of Education I asked Doctor Gwyneth Hughes, Senior Lecturer of Higher Education..

How much resistance do you think is there amongst academics who don’t really see it as their role to develop the aptitudes and abilities of their students but to teach them content and get them through exams and various other forms of summative assessment.

Gwyneth Hughes

Yes I think that is crucial. Academics feel passionately about their disciplines. So they feel their role is to impart that passion and interest and rigour and high standards of their discipline to their students. So I think it is a challenge. But there is increasing awareness and we have a wide body of students and we’re not teaching students to be specialists in our disciplines and that students might go on and have a career that’s completely unrelated to the discipline. But there are skills in every discipline, for example in education, there are writing skills, critical skills, interpreting arguments and relating those to practice, now that’s a very important thing to do, to relate theory to practice and that’s something you can apply to pretty much every walk of life even if you didn’t stay in education. And that applies to other disciplines. So I think this focus more on maybe matching up generic and disciplinary skills and having these conversations about, not just the content, but what might be taken forward into the future

Kim Catcheside

How do you get that message over to academics that might not value other skills?

 

Gwyneth Hughes

Well I think that’s very difficult, this is why I’m frustrated. Views are shifting but very slowly. I think we are getting new lecturers who now do a certificate in teaching and learning and get a qualification and become fellows in the higher education academy and they are looking at these kind of ideas about learning in the longer term and developing students, that it’s not just about the specialists and the subject discipline and those who are going to go and be the top specialists in their fields.

 

Kim Catcheside

At the University of Exeter, Tim Quine is Professor of Earth System Science and Associate Dean for Education in the college of Life and Environmental Sciences.

Tim Quine

Well there was a time when people just came into universities and just started lecturing but now we have in service training programmes so that academics are taught how to teach, they’re taught how to teach in collaboration with each other so then on programmes in which they’re comparing notes, our education advancement group will tell them about the latest pedagogic advancements. So they’re not just left to flounder we monitor not only their success in research and granting and publishing but also in terms of their involvement in education. How well their modules are being received, how well students are retaining on those modules. We look at that on module level and on programme level. So across the piece we are careful to monitor that students are getting a good deal.

(Sounds of walking into a room)

Kim Catcheside

Tim is also involved with Collaborate, a major research project, supported by Jisc, which aims to guide and inspire and offer useful digital tools for academics and students to integrate employability skills in all aspects of assessment.

Richard Osborne

My name’s Richard Osborne and I’m the Project manager for Collaborate at Exeter University and at the moment we are in our brand new forum building and outside one of the exploration labs. And we’ll just go in and have a look round.

So we’ve got two exploration labs in our new forum. This is the surface table room and here we have 10 surface tables all about 60 inch in diameter and these are multi touch interaction tables were four users can simultaneously log in and collaborate and create together.

Kim Catcheside

These surface tables are similar in appearance and scale to snooker tables and their interactive surfaces – they are in effect massive tablets and mean that students and staff to work together on research or a presentation –this is central to the principles of Collaborate. Tim Quine

 

Tim Quine

One of the key skills for a student leaving with an environmental science degree is the ability to use modern software to work in, with specialist data such as remote sensing imagery, GI’s: geographic information systems and we’re very keen that the students not only work independently but also work collaboratively in the use of this software. So I was involved with the committee that commissioned the room that we’re sitting in here with these exploration tables and we, we envisioned students coming into this space, poring over aerial photographs comparing notes, interpreting these together and then going out into the field to explore what they’ve seen digitally and then returning to analyze the data.

Kim Catcheside

Liz Dunne is head of project development in quality education enhancement at the University of Exeter and director of Collaborate. On one of the surface tables, she’s showing us the Radial diagram, which is at the heart of the project.

Liz Dunne

We have six dimensions that we’ve been looking at in this model and those dimensions range from collaboration, to problem solving and data, to audience, to time, to peer review and to structure. So if I just take another example, audience is very, very important to employers. If again I take the example of the traditional assignment, an essay, that’s usually written for your tutor, your academic member of staff. Whereas employers say, well that’s fine but you need a much broader audience so we’re thinking about audiences of peers, we’re thinking about audiences through technology using film and video podcasts perhaps. We’re looking at employers as a specific kind of audience; we’re looking at broader initiatives beyond even the employers that link into the particular subject areas. So different groups and working parties perhaps that would be interested. So constantly thinking about what the range of audiences could be for that particular task.

Kim Catcheside

With us is, Samson Matthew a student at the University of Exeter Business School, where he’s studying Management and Marketing.

Samson Matthew

I think it would it would be much more powerful to graduate with a degree that kind of assesses your ability to lead people or your ability to communicate to people along with your marks. Rather than just strictly, oh I graduated with a 1st. I know way too many people who sit in their room all day and they can study and get amazing grades, but the moment they enter the job market, you know employers are kind of, they’ll hire them on their great grades. But you know, they perform total different in the work place itself right. So it’s definitely, so maybe the model isn’t perfect now, and maybe it will take a long, long time for it to really be incorporated into education, but I think eventually it will reach a level that it becomes really, really powerful for both students, educators and employers.

Kim Catcheside

The radial lines in the diagram are intended to act as inspiration as well as guides or prompts for work-integrated assessment in course design and delivery – if they were all the same length the diagram could resemble the points of a hexagon.

Tim Quine

Looking at the model that were rolling out, the Collaborate project, I wouldn’t see us saying there’s an optimum shape of hexagon – that everyone has to have a big hexagon a long way out on each of the dimensions but I would be encouraging my programme director’s and my directors of education to look at our programmes in the round and say are we stretching along each of these axes, when we look at our programme in it’s entirety. I think the individual modules may well focus on the different aspects on that, those six dimensions. As I said in my last steering group, if I had to identify six dimensions along which students would feel most nervous there the six dimensions we have here and if we went straight into pushing out along those dimensions we run the risk of having some very nervous and very agitated students. Satisfaction levels might not necessarily increase.

Kim Catcheside

To illustrate the way Collaborate’s radial design might work in practice, we’re joined by Holly Golden, a recent graduate in History and International relations at the University of Exeter who is going on to do her masters in Politics research.

Liz Dunne

Her particular academic tutor was doing something that was innovative and very engaging for students. Ok so if you’d like to talk just a little bit about the dimension of the audience

Holly Golden

In our assessment as well we actually had to think about who our piece of work was being aimed at, It wasn’t just Clare our tutor who was marking it, she talked about how we should think about sort of, ours was policy analysis, so it was sort of which government area we were talking to, whether it was young people, old people we actually had to think about that and how that would actually reflect in our writing.

Ok so if you’d like to talk just a little bit about the dimension of the audience

Well we looked at a lot of different audiences, not only in the actual assessment that we did but also during the actual module seminars. We not only got interviews and people coming in from the outside, so we obviously learnt what they saw as their audience and they talked about those sorts of aspects, but in our assessment as well we actually had to think about who our piece of work was being aimed at, It wasn’t just Clare our tutor who was marking it, she talked about who we should think about sort of, ours was policy analysis, so it was sort of which government area we were talking to, whether it was young people, old people we actually had to think about that and how that would actually reflect in our writing.

Well Clare was quite specific about that. She didn’t want a long academic essay, where obviously it’s just standard that’s what normally happens. It was actually she said it would be if you were actually a policy analysis advisor. So she said, obviously it would be quite a short piece of work, so she limited it to I think 1,000 words. But obviously it was mainly bullet points which I thought was quite interesting rather than writing a long essay, it was bullet points and advantages and disadvantages and she did it that way as if we were actually in the workplace because she said when you put it on someone’s desk they’re not going to want to read loads of pages they’re just going to read the front page and read it from that.

And I know you worked in collaborative groups, so would you like to talk a bit about collaboration?

We actually had to run sessions instead of having a solid structure where you had a lecturer sort of speak for an hour and then you had a group discussion, she actually left us and said, put us into groups and said, you all run a session. So we obviously had to think of a lesson plan for ourselves and you got a lot of ideas thinking which I think worked really, really well which was brilliant

Kim Catcheside

Unfortunately we don’t have time to listen to Holly’s entire experience of the work integrated assessment to her course, but it is available as a separate sound clip on the Jisc On Air site.  But just picking up here on Holly’s enthusiasm for peer review at Exeter is mirrored at Strathclyde University where David Nicol, supported by Jisc, made his own study of peer review. Design Engineering students were asked to comment on the work of other students. The quality of their work improved enormously as a consequence…

David Nicol

When I asked them what happened when you did this? What they said was very interesting – they said the work I produced myself was my internal standard, so whenever I saw something in other people’s work that was different from mine and might be better, I reflected back and though how it might improve my own work.

Kim Catcheside What about digital technology

David Nicol

Well that peer review study could not have been done very easily without the technology  and it added value. So pure review software automatically distributed two assignments to every student to have it review and after it was reviewed it sent the reviews back to the students – so it  saved the teacher a huge amount of time – it was anonymous so it saved time in terms of taking names off and collating feedback and redistributing to the students, so it was all done automatically by the software – so an incredible advantage to have the software and actually pretty hard to do on a regular basis without the software

Kim Catcheside

Giving students valuable feedback so they can understand the curriculum and their progress through it; to consider how to deliver integrated assessment that prepares them for a working world, can be enhanced by the use of digital technology… and it doesn’t have to cost any more according Gill Ferrell, independent education consultant, working with the Jisc on the Assessment and Feedback programme

Gill Ferrell

I don’t think that people necessarily need to invest in new technologies in order to optimise what they’re doing in assessment feedback, they need to look at the infrastructure they’ve got and ensure they’re using it optimally, They might have a robust central infrastructure but if every school and faculty use it in a different way then you’re not really getting the efficiency from it. In terms of what students and tutors are using for direct engagement with one another than trying to make the most of relatively simple and familiar tools seems to be where people are making the most gains. You don’t have to invest in something that’s new and different. Simple tools like podcasts to give feedback like all sorts of collaborative social networking tools that exist and are in day to day use that can help with peer groups working together and giving each other feedback are examples of things that are easy to do and are showing real benefits.

We often talk about feeding back and feeding forward and the difference between assessment of learning and assessment for learning and where people are really trying to put the emphasis is on developmental feedback on actually improving learning so that by the time you get to this very formal summative assessment at the end of the year, you know all the tutoring that you have has actually fed into that.

It’s about thinking about the educational principles that underpin the learning experience at your institution and it’s about designing learning experience that fit with those principles and then finding simple tools that can actually enhance that learning experience.

Kim Catcheside

And this is exactly what the collaborate team are proposing…for teaching staff and students there are also the Top Trumps – Stuart Redhead is project officer of Collaborate

Stuart Redhead

We have to make sure we use technology that students are a, happy to use. It’s intuitive, looks good, we need to make sure that this is something that students are familiar with. They come along with their ipads, they come along with their laptops, they’re used to using certain types of technology and technology that quite often looks good. So we need to make sure that in this project we allow them to use technologies that they’re happy with and that also fulfils the remit that we need. Hence were hoping that with the technology ‘Top Trumps’ to produce a package that students will recognise many of the technologies but also to be able to discover new ones and be able to use them innovatively and take that forward into their future careers.

Richard Osborne

The Top Trump cards is our way of bringing technology into the mix here. We’re developing a series of cards using the Top Trumps metaphor, which basically take off the shelf technologies for the most, things like blogger, things like pod casting tools like sound cloud and they rate them according to the model. So we might say blogger is really good for structure because it gives you that diary based form and perhaps good for time in that context too because it works over a period of extended time. So we’re rating these things along these dimensions and then people can see how a specific technology can help them explore different ways of running an assessment.

Kim Catcheside

If you would like to explore the other projects that Jisc is supporting to enhance Assessment and Feedback with technology please visit http://www.jisc.ac.uk/whatwedo/programmes/elearning/assesmentandfeedback

One thought on “Driving change in assessment and feedback

  1. Pingback: Driving change in assessment and feedback at JISC On Air

Comments are closed.