The Impact of Assessment on Teaching & Learning: How Does Assessment Shape Learning?

[00:00:07] Speaker A: Welcome to season 12 of Talking ELT. [00:00:09] Speaker B: Ooh, exciting. [00:00:11] Speaker A: In this four part series, we explore the powerful and sometimes problematic relationship between assessment and teaching and learning in elt. [00:00:20] Speaker C: And then they asked me the same thing. Why not 31, why not 32, why not 28? And I wasn't, I couldn't really explain. I had this gut feeling that, well, you weren't as good as the previous one. But I cannot really put it into words. [00:00:34] Speaker B: We can all point to examples of what we think is an influence of testing on teaching and learning, but then how you go about researching it is quite challenging. [00:00:42] Speaker A: I'm joined by Joe Soke, freelance teacher, trainer and contributor to Oxford University Press's position paper, the Impact of Assessment on Teaching and Learning Creating Positive Washback. And Nate Owen, senior research Manager at. [00:00:57] Speaker B: Oxford University Press, where you can say, okay, this is how you study it, how you exemplify it, and this is how you prove its existence. [00:01:05] Speaker C: Sometimes I feel that it's for their confidence or for them to feel that they are fighting or learning towards something. [00:01:14] Speaker A: Together they dive into topics like assessment, literacy exam reform, alternative assessment methods, and the evolving role of AI in feedback. [00:01:24] Speaker C: And testing because there's positive washback and negative washback. Hard to explain which one is which. And as you mentioned, so complex that you get mixed up by like, not knowing. Okay, so which part affects which part in the. In the entire cycle? [00:01:45] Speaker B: We're all language based professionals. We're involved in teaching and assessments. Of course, we're very, very passionate about language. But the reality is that many of us students are not particularly passionate about language. They see it as a tool and they see a test as a passport. [00:01:59] Speaker A: It's a lively, thought provoking series with real world relevance. Plenty of ideas to take back to your teaching context. Let's jump in. So thank you so much for joining us. This is such a joy to have you both here talking about this subject which so many people are interested in, the impact of assessment on learning, the relationship between the two. I think we're going to have a really good time just delving into this topic with two people who know what they're talking about, which is. Which is hopefully. Yeah, well, we'll judge that at the end. [00:02:43] Speaker B: We'll assess that or we'll learn it. [00:02:47] Speaker A: Okay, so we're going to start off, I think this whole series of podcasts is going to focus on the relationship between assessment and learning. And what I want to start off with is you to kind of introduce yourselves by telling us, telling the listeners what took you on a Journey towards being interested in the relationship between assessment and learning. How did you get to this place where you are now? Who wants to start? You start. [00:03:19] Speaker B: All right. Yeah. Well, I think I became a language teacher around about 2007, and my early impressions of teaching were very fragmented. I'm sure the industry hasn't particularly moved on. Massively, very informalized, not a lot of job security for language teachers. And so we're just kind of thrown into a classroom a little bit. You have new students turn up on a Monday, some students leave on a Friday, new students on a Monday. And as a result, you don't really get a chance to build up much momentum or necessarily get to know the students over a long period of time. That can make classroom dynamics very challenging. Until I was given an exam class one time, and I think it was a first certificate course that I taught, and I had the same students for 12 weeks, and they all came on time, they all participated, they all came every day. They were all invested, they were all motivated. It was radically different to the sort of very, very informal, ephemeral nature where people would sort of come and go a little bit, and, you know, in the regular classes, people might turn up a little bit or they might wander out early. And so the dynamic was just radically different. And I thought, oh, I quite like this, to be honest. And so from that moment on, I actually sought out the exam courses and said, I'd like to teach more of these, please. And so that's how I got into assessments early on. And I think it was because I noticed there's a difference here, and this seemed to be perhaps an unfashionable thing to say, a positive instance of assessments on classroom dynamics because they were invested, because they wanted to be there, and they knew there was a specific outcome. We were all working to the same goal, we knew what we were doing. Everything we did had a purpose. And of course, in an exam class, the first thing you do, of course, is you go through the exam and you say, look, this is what it all looks like. This is what you need to do. And these are the sorts of activities we're going to be doing. We won't just be doing rote learning, and we won't just be doing practice exam after practice exam, but all the activities will sort of be related in some way. If we can extrapolate what the skills are, then we can do activities related to the end points, which is this exam. And if you're clear about what you're doing, everyone's on the same page. Everyone buys in and then you're away and you're good to go. And I think that for me was how I thought, okay, this can actually be a good thing. So that's my start. [00:05:55] Speaker A: Interesting. [00:05:56] Speaker B: Nice. [00:05:57] Speaker C: Well, I think we kind of started teaching at the same time because I started teaching in 2008, but my interest in assessment came from a different angle or a different starting point because I always hated being assessed when in school. I just hated all kinds of assessment. I didn't see the point. I didn't see why we were doing what we were doing and what the point was of each test. Also, I felt that they didn't really prepare us for the tests. I kind of felt throughout my school years that this is not how it should be done. I didn't know at that point what I wanted. But now as an ELT professional, I can put it into words, I can see the system, I can see what I missed. And back in Hungary, I have a university course, Assessment and Feedback in elt. And that's how we start every single course with my students that I ask them what they felt when they were back in high school, whether they liked being assessed or what they didn't like. And I actually want to focus on the negatives because from that we can build up what they actually had in mind and what they expect. And that was why I got interested in this whole thing. That I knew somehow that this wasn't right and that we should assess skills differently and more authentically, but I just didn't know. Now I know. Well, sort of. [00:07:28] Speaker B: No. Interesting, really. I think all teachers can relate to that. There's a sense of we must right the wrongs of our own education as we experienced it. I remember going back to high school and trying to learn German and French in secondary school, in 35 classrooms with 35 and one teacher at the front. And you get maybe one or two hours a week. And even as a 13 year old, it's clear this, this isn't working. This isn't how you do it. I think as an adult, you think as a language teacher you want to put that right somehow. [00:07:58] Speaker C: Definitely. But sometimes we're locked in the system in a way. And that's the most difficult part of it that I know. Now that I'm on the other side, I'm not the student anymore, but the teacher. I can see how teachers are struggling and how they are trying to make all of these angles work together. And it's super difficult, but still we need to make a change. [00:08:21] Speaker A: But I think it's such a good idea to get people to focus explicitly on the things that they struggled with as students because otherwise there is a risk that we just automatically replicate our own experience of the classroom because that's what we saw our teachers doing without really thinking about which elements worked and which didn't work. [00:08:42] Speaker C: Also, there's another danger in it, I think. So I try to play this game cautiously because you can go from one extreme to the other. So I also when for example, these students of mine, they are full time students, so like 20ish, I think. So they are pretty young and there's this danger that they might just fall to the other extreme and say, okay, so if my teacher did this and I didn't like it, I'm going to do the exact opposite. But that's not always the answer to the question. So we need to find these fine lines and the gray areas of assessment. [00:09:22] Speaker A: Yeah, I mean, for me, the experience which triggered my interest in assessment and learning was teaching in Sri Lanka and I was teaching just a regular class and I had to give termly semester grades for the students speaking and writing for their language, but particularly their speaking. And I would do this just very impressionistically, not giving it much thought. [00:09:50] Speaker C: Yes, yes, I triggered something happening. [00:09:53] Speaker A: Anyway, this student challenged me. You know, last term you'd given me an A on this and now you've only given me a B plus or something like that. And I hadn't given it that much thought. She said, why? What do I need to improve? She said, because I'm desperate. I need to pass ielts in order to get to Australia or something. Like her whole life plan was based on this. And here I was giving her negative feedback to a drop in grades and I had no explanation. I didn't know. And so suddenly, you know, the penny dropped. I need to understand exactly how I'm assessing. I need to be able to give feedback on what she can improve in order to be helpful. Otherwise I'm being the opposite of helpful. As a teacher. [00:10:41] Speaker C: I had a very, very similar experience and that's when I started using rubrics in my classes because I had the same experience. I was doing some sort of, I think a presentation skills class and. And I did the same thing. So like I felt that this is a bit weaker than the previous one was. So okay, I'll give you 30 points out of 40, but. And then they asked me the same thing. Why not 31, why not 32, why not 28? And I was, I couldn't really explain. I had this gut feeling that, well, you weren't as good as the previous one. But I cannot really put it in the words. I cannot specify why. So I came to the conclusion, okay, I can now see the point of rubrics and making it clear why you got certain point levels or points for certain things, and that's how you can get feedback and continuous and consistent feedback. [00:11:38] Speaker A: Yeah. Okay, so I'm gonna. That was all really interesting. Now I want to kind of get into the. To the meat of the topic. And what does washback mean? Or backwash? Is there a difference between them? What do they mean? I'm kind of looking at Nate now to give us some kind of explanation. [00:11:56] Speaker B: I'll try and give you the official definition, such as it is. I mean, this. I think the term washback, I don't remember if it was exactly introduced by Diane Wall and Charles Elderson, but they certainly popularized it. And so, generally speaking, their definition is washback refers to the impact of testing on teaching and learning. That's a kind of one sentence definition. If you expand that a little further, they talk about the impact of what is learned and what is taught, how it is taught, how it is learned, how it is communicated. And so it became very clear that this was a multidimensional, multivariate phenomenon that was maybe a little bit nebulous. Quite hard to pin down a little bit. I'm still not entirely sure within the literature that it has been codified in such a way that we can say, okay, this is how you study it, and this is how you exemplify it, and this is how you prove its existence, even because we all know as a thing that seems to happen, and we can all point to examples of what we think is an influence of testing on teaching and learning. But then how you go about researching it is quite challenging. And I think that's one of the reasons that I think the term has maybe fallen out of use a little in more recent times. [00:13:19] Speaker A: Yeah, yeah. I mean, it's. Yeah, it's very interesting because you write Diane Wall and Charles Alderson. I think they started off with a kind of assumption that it was so obvious that people teach to the test, and therefore you change the test, you change the teaching, and that changes the learning, et cetera. People are motivated by passing that. It was almost a truism that didn't need proving. And then they engaged in a project where they were changing the exam. I think, again, it was Sri Lanka. They were changing the design of an exam in order to improve the teaching approach. And what they found, surprisingly, was it didn't change the teaching approach. And then they kind of, from this, identified other factors. The key factor in this case being teacher beliefs. So if a teacher believes this is the right way to teach, they will teach it, even though the exam is pushing them in a different direction. So it was starting to recognize there are lots of different factors that shape the way we teach. It's not as simple as one thing. Yeah, and you're right. Since then, there've been a lot of research projects which have investigated that, but probably no clear answer because it is such a complicated relationship between the different factors. [00:14:47] Speaker C: And it's. Yeah. Then because there's positive washback and negative washback, it's hard to. Hard to explain which one is which. And as you mentioned, like, it's so complex that you get mixed up by not knowing. Okay, so which part affects which part in the entire cycle. [00:15:09] Speaker A: So people who are new to this positive washback versus negative washback, the way. [00:15:15] Speaker C: I usually explain it is like negative washback would be to me. But Nate, correct me if I'm wrong, is the one when you're teaching to the test, or somehow you let the test influence your teaching method so much that you completely change your teaching methods to make students pass, and you are not teaching in order to improve them or help them to learn anything. You are teaching them to pass. And. And to me, that's negative washback that you are not. Nothing useful actually comes out of it. And the test kind of stifles you so much that. That you are just. You keep that in mind completely. While positive washback would be that you are trying to focus on transferable skills and how the test can help you improve students on the long run. [00:16:17] Speaker A: Right. [00:16:18] Speaker B: To me, then, yeah, it's interesting picking up on this phrase, teaching to the test. At the IATEFL conference earlier this year, there was the debate, the OUP debate, which was around whether or not standardized assessment could have a positive impact teaching and learning. And Barry o' Sullivan was speaking for the kind of four motion. And he picked up on this phrase, teaching to the test. And I remember something he said because it was quite interesting. He flipped this on his head a little bit and he said, you better teach to the test, because if you don't, your students won't pass and their parents are going to be really angry with you. [00:16:53] Speaker A: Yes. [00:16:54] Speaker B: And I thought that was an interesting way of looking at it. [00:16:57] Speaker C: Well, that's how the system keeps you in this locked position, because, yes, of course you want to pass the Test if you want to study in Australia, if you want to like prolong your visa. But then that's how the test can also have a. Sorry. [00:17:13] Speaker B: Yeah, no, no, I mean, I think, yeah, it means that we need to unpack this phrase teaching to test a little bit and find out what's the negative thing that people are really objecting to when it comes to teaching to the test. And I think it really comes down to this almost notion of rote learning that you're describing, whereby let's say we're teaching an exam course and let's say, okay, today we're doing writing, today we're doing reading, or today we're doing listening. And here's our listening activities. These are the listening activities in the test, therefore we're going to do these. Oh yes, and then you're just doing tests. Preparation, practice, again and again and again in the mistaken belief that simply repeating something long enough will make you good at it. That is, yes, a very clear example of what I would describe as negative washback teaching to the test that nobody. [00:18:00] Speaker C: Likes practice test after practice test. [00:18:02] Speaker A: And yeah, I mean there are, there's a fairly obvious kind of situation where you might have a test, particularly in a school, national school system, which doesn't have a speaking component. You know, so it's just a lot of grammar, vocabulary, multiple choice questions, some reading comprehension, possibly a listening comprehension. And so the focus of the teacher is on those skills. Even though the students may actually want to develop their speaking skills, it's no longer, it's not important. So that's a fairly obvious curriculum problem. And there are a number of cases where they've introduced speaking tests, international exams, and with the expectation that it'll have a positive impact, but without the support to the teachers who are not used to teaching those speaking skills, who lack the speaking skills themselves. Which is why it's never just the introduction of the exam itself that changes things. [00:19:03] Speaker C: Oh yeah. And also when you're teaching a skill, for example, when you're teaching speaking, it's, it's such an interesting question when. Because you are not. So I sometimes have to tell my in service pre service teacher trainees that teaching speaking is not trying to help students complete a task. So like the exam task, teaching speaking is how to teach the skill which has sub skills and different elements. And we need to help with authentic examples and activities. But all I can see is that they have the speaking tasks in their minds and they know like okay, yeah, on this exam, task one is talk about yourself, task two is discussion and this is how they see speaking as a skill, which is a completely different thing in real life and should also be reflected in exam design. [00:20:04] Speaker B: I think this is the challenge as an assessment professional now, is when we design assessment tasks, it's not only about making the tasks available to teachers and learners. You've got to actually communicate to all stakeholders why the task looks like this, what's it really trying to get at? What are the kind of skills embedded in this? And if you make those publicly available, you communicate those, then you can start, hopefully. And this is why we do it, to encourage that little positive washback whereby we say, well, actually, if you're doing any kind of activity that relates to these skills, you are in effect, preparing people for this particular task. That's what we try and do. Hi, I'm Nathaniel Owen, Senior Assessment Research and Analysis Manager at Oxford University Press, and I just wanted to point you in the direction of our fantastic position paper, the Impact of Assessment on Teaching and Creating Positive Washback. Our team of distinguished experts go into detail on the concept of positive washback, as well as sharing some recommendations to help schools and policymakers foster positive watchmaking and exploring how emerging technologies are reshaping the way educational institutions approach testing and assessment. Go ahead and download the paper today by clicking on the link in the description and enjoy the review. [00:21:21] Speaker A: So should we think about the impact on learner motivation? Because I think for a lot of people, the role of tests and exams, one of their roles is to motivate the learner. And I found this both with kind of high stakes exams, but even your class tests. Why do you give the class test to motivate the learner. Does I mean, is there some weight to that? Does it motivate learners? [00:21:49] Speaker C: I find it like, yeah, very often when I start a new course in a language school or when I have private students, they say this exact same thing. I want to take this exam because I want to be motivated, I want to have a goal. I, to be honest, sometimes I feel that it's for their confidence or so some for them to feel that they are fighting or learning towards something. However, I don't always think an exam is necessary in every single case, but they feel that that's a clear expectation and that's something that they can clearly see. There are clear rubrics and checkpoints, so it makes it easier for them. But I think to motivate them, we can set other goals as well, but it's a clear one. [00:22:45] Speaker B: Sort of goes back to the old division between intrinsic and extra Motivation, doesn't it really? I mean, we're all language based professionals. We're involved in teaching and assessments. Of course we're very, very passionate about language. But the reality is that many of our students are not particularly passionate about language. They see it as a tool and they see a test as a passport. And maybe they want to be doctors or engineers or accountants or whatever else and they know that in the international workplace they're going to need English. So okay, English is my passport, English is my tool. So I'm not necessarily motivated to learn about all these kind of cultural artifacts associated with English, whether they, be they British or American or whatever. They just want to learn English for what it allows them to do in the future. Yeah. So we have to sort of try and take on board where they're coming from as well. A little bit. [00:23:36] Speaker C: Definitely. However, it's interesting, I do sometimes do business English courses and business English students are an interesting breed because sometimes you get the ones who, just like you said, they want to, they want to fulfill their roles in communication, but they know that for that they don't need anything, they just need to get the deal done. And they, they know that it's enough if they are understood. They know what to expect from their communication partners, that it's enough to just get it sort of right. Not necessarily grammatically, in a grammatical correct way. However, you also get those students who know of the existence of a business English exam and they say, okay, I actually want to do it right, so I need this exam. So their, their intrinsic motivation, as you said, it's coming from different place. Some, some of them, one, they are very task focused and they know that to be honest, I don't really need the exam for that. I just need the signature on the contract. While. While others have different intrinsic motivation probably because of their own beliefs, because they value an exam and that's why they are looking for such an exam. [00:24:58] Speaker A: Yeah, I mean, I think one of the other roles it might play is that it's very difficult to monitor your own progress in learning a language. I mean, it's difficult for anybody to see whether they have achieved anything. And so, you know, a test or an exam gives you that sense that you have achieved something. [00:25:16] Speaker B: It's a good benchmark, usually against themselves. Especially these kind of low stakes assessments, classroom based assessments. I always found that students quite like them to benchmark against themselves and think, okay, that's, that's what I need to work on. That's what I'm kind of doing well. Yeah. And they're not necessarily comparing themselves to their partners. Yeah. [00:25:35] Speaker C: That's why these gamified tests can be a good idea because they can repeat the test again and again and again so they can see progress. And sometimes with high stakes exams, it's not so easy to repeat them week after week. [00:25:53] Speaker A: Not so easy as impossible. Yeah, yeah, yeah, no, that's right. Okay, so what are some of the common misconceptions that some people have about testing tests in the classroom? In language education, I would say a. [00:26:12] Speaker C: Common misconception on both sides is that testing is terrifying and should be terrifying. So from the teacher's side, I think it's a kind of misconception that testing is used as punishment sometimes. And then on the student side they are afraid of tests because of the same reason that it tends to be used as punishment. And I think that's the worst rap testing can get because that's not the purpose of testing. [00:26:45] Speaker A: Yeah, I've heard teachers say that test, it's a four letter word which you mustn't say, but then they say quiz instead, which is also a four letter word. So, yeah, not sure if they've escaped it, but that's a lot of what the kahoot kind of quizzing is. It's a form of testing, but feels like it's not as formal. [00:27:07] Speaker B: Yeah. [00:27:08] Speaker C: Because it alleviates all the pressure. I remember, gosh, when I was in high school, we had a literature teacher and, and somehow we were trying to judge based on her footsteps that we could hear from like outside whether she's going to give us a test on that day or not. And then she came in, slammed the door and we instantly, like, I had this feeling in my stomach that, oh no, she's going to say the four. It's actually not four letters but like, I don't know, like get a piece of paper, a blank sheet of paper and then it was on. And that feeling, it just stays with me forever. And I didn't know even at that point what's the point of all this? Why are we doing this? And yeah, I don't want that to be felt in the classroom. [00:27:59] Speaker A: Yeah. [00:28:01] Speaker B: Well, I think it's an opportunity for me to get on my soapbox here because I think as an assessment professional who works in high stakes assessments, the term that I think is most misunderstood is standardized testing. Standardized testing is something that gets a very negative rap. And I'm sure anyone listening to this or watching this will immediately have an aversion to any concept of standardized testing. But then we have to think Again, just like we unpacked teaching to a test a little bit early, we have to unpack what we mean by standardized testing. Because all testing is standardized to a certain extent. If you're giving everybody the same test, that's a form of standardization. If everyone is being marked on the same rubric, that's a form of standardization. If everyone is in the same classroom at the same time taking that test, it's a form of standardization. So again, and this is not something that has been culturally imposed by the west on the rest of the world, it's something that actually is a Chinese invention. Standardized testing was invented back in the 8th century in China with the imperial civil service examinations. So the concept of one single examination paper, the concept of invigilation, the concept of a single exam at a single place in a closed exam room, is actually something that was invented in China and that the west learns from China and that many of the origins of modern testing in the west came from civil service reviews that civil servants from the west undertook around the world. How do people, you know, recruits, train and certify civil servants around the world? And so civil servants in Germany and America and the United Kingdom learned about what was going on in China. But wow, this is a really good system, so we can actually learn from this. And so all of these things that we take for granted came from, you know, we sort of caught up maybe 1200 years later. But what people, I think what people object to when it comes to standardized testing is what happened really in the 1970s when the notion of standardized testing was probably taken over by the psychometricians and the notion of the standards went from a kind of criterial performance and describing what people can do in the real world, which is how criterial reference testing was originally envisaged. And it became about cut scores and spreading people out. And so there was this strange amalgamation of norm referenced and criterion referenced testing that happened in the 1970s. And now what we call criterion reference testing with cut scores is not really criterion reference testing at all. It's a norm referencing with maybe some reference to external criteria pasted on the top, such as the common European Framework of reference for CFR for example. But no, I mean the concept of criterion reference testing was used originally to describe people who were taking exams related to machine operation in the 1960s. This is how people talked about criterion testing. And if you're learning how to operate machinery, what does the test look like? It's an actual instance of you operating the machinery overseen by somebody like a driving test. I Think there's a lot we can. [00:31:20] Speaker A: Learn and if everybody passes, that's a good thing. [00:31:22] Speaker B: Yes, exactly. Yeah. So that's criterion referencing. So there's no, there's no onus on you to separate people out and say, okay, let's compare people against each other. If everybody is able to operate that machinery. [00:31:35] Speaker A: Yeah. [00:31:35] Speaker B: Then everybody is certified. [00:31:36] Speaker C: And it shouldn't be a bad thing because sometimes you now get these ideas that, oh, a five or a top score is never given to anyone because, come on, you cannot just obtain that high level of knowledge. No, that's not, not for you. [00:31:51] Speaker A: But it is. I mean, it's part of the, the national education system. Is that it is separating people out. [00:31:58] Speaker B: Yes. [00:31:59] Speaker A: I mean, it would be a problem, I think, if everybody got top school. [00:32:03] Speaker B: Yeah, it would. Because the whole point of a test is to be able to make decisions about people. [00:32:08] Speaker A: Yes. [00:32:09] Speaker B: And you need to. Well, you need to be able to do that rapidly, efficiently. Universities want to know, okay, who do we accept? Who do we not accept? Of course there are other means of doing it. [00:32:19] Speaker C: I totally get it. [00:32:20] Speaker A: Yes. [00:32:21] Speaker C: Assessment is the main reason or point of assessment is to make decisions. But then, like, why is it a problem that if, like what if everybody knows the language? Well, of course we need to define what well means, but why is that a problem? It's like these are two separate things. One is like trying to assess their language knowledge and then trying to decide if they can get into this educational institution or not, or if they can get this job or not. But they might know the language. Well, all of them. But perhaps they need further and additional things to qualify to this. [00:33:05] Speaker A: That's quite a nice teaser for what we're going to come back to in future episodes for this series because I think this whole question of grading and its role assessment is. I think they've kind of done a, a nice first look at relationship between assessment and learning. Positive washback impact, relationship between them, why it's important to understand testing properly. So I think that's a wrap on the first episode. [00:33:37] Speaker B: Fantastic.