Episode Transcript
[00:00:06] Speaker A: We do risk over assessing people. So here I am, an assessment specialist saying, no, assess less, a little bit.
[00:00:13] Speaker B: There should be a purpose, but also we want to make decisions and we should always be aware of what kind of decision we want to make and why we're doing what we're doing. We tend to think that standardization is the worst thing ever because it keeps or puts people in boxes and sets everything. But yeah, we need to somehow keep ourselves up to a standard to see how we are measuring up or how we exist in society.
[00:00:39] Speaker A: We have to accept that failing a test is not necessarily unjust if you.
[00:00:45] Speaker B: Communicate everything fairly and you communicate your evaluation fairly. I mean, I failed my first driving test, but I remember why I wasn't ready yet.
[00:01:03] Speaker C: Welcome back to this next episode. We're looking at the relationship between assessment and learning. We kind of explored in the first episode about the nature of that relationship. Today I wanted to look at what that means for a teacher. So if you're a teacher and you're trying to look at making a positive relationship between assessment and learning. So one of the questions that we might have is what do teachers need to understand that assessment to help them as teachers helping their students?
So it's kind of the assessment literacy question.
Is this a question you've given any thought to, Nate?
[00:01:42] Speaker A: Certainly, yeah. I mean, the most obvious thing that springs immediately to mind of any assessments, whatever it is, whether it's a high stakes assessment or whether it's a little classroom assessment, every assessment should have a purpose. And that purpose should be communicated clearly to students so they know why they are being tested. That in a nutshell is probably the most important thing anyone could know. Yeah.
[00:02:05] Speaker C: And to understand what clearly what the purpose is, what's it going to be.
[00:02:09] Speaker A: What'S it going to do, what is, what are the results going to be used for? What are we going to learn from this?
[00:02:14] Speaker C: For example, just to kind of clarify.
[00:02:16] Speaker A: What you mean, I mean, diagnostic assessment is a brilliant one and diagnostic assessment, again, another term which we're just throwing out there. But sometimes people give tests at the beginning of a course and they do that because maybe I want to find out where my students strengths and weaknesses are.
So that should then lead very directly into whatever it is that you're doing in the classroom. But again, it shouldn't just be a test that you're given at the beginning of the course and then nothing really happens with that.
Or that maybe you only look at the scores or you only give the scores to the students. It should be very much the case that the test is designed in such a way that you can see, okay, which questions are the students may be a little weaker on? Which ones are they doing well on? Okay, these questions are about these particular language points or these particular skills. So that's where we need to focus on a little bit more. So it's a very, very clear instance of subsequent decision making on the basis of what you've achieved in that test.
[00:03:14] Speaker C: Which relates to what we were saying at the end of the last episode about sometimes the purpose of a test is actually to separate out people.
[00:03:23] Speaker B: Yeah.
[00:03:23] Speaker C: Because it's the gateway to next step in their education or something like that.
[00:03:28] Speaker A: Yeah, yeah. I mean, so that's a placement test. I mean a placement and a diagnostic test. Unfortunately, there can be too many different tests because again, in the worst case scenario, what you'll find is that you have an incoming cohort of students. First of all, they're given a placement test and that sorts them into their respective classes. Then in those respective classes they're given a diagnostic test and then, okay, now we know what we need work on. And then a week later, oh, you're given a progress test and it's like, oh, after that, or you're given, you know, maybe another progress test or a midterm assessment or an end of term assessment.
We do risk over assessing people. So here I am an assessment specialist saying no assess left a little bit.
But yeah, so if you can very clearly say why you're assessing people and what your assessment is for, then it is justified.
[00:04:12] Speaker C: Yeah.
[00:04:12] Speaker A: But yeah, we must be careful not to just assess for the sake of assessing.
[00:04:16] Speaker C: Mm, definitely join your. I think you mentioned you run a module in your teacher education on assessment.
What, what, what's the most important things you try and get across in that?
[00:04:28] Speaker B: Yeah, definitely types of tests for sure.
Getting this across, this message, the main message across as well, that there should be a purpose. But also we want to make decisions and we should always be aware of what kind of decision we want to make and why we're doing what we're doing. But also I want to clarified other terms just to introduce more terms validity and reliability because.
And that's how I can always link it back to. Usually on our first occasion we talk about why they hated assessment in their high school or primary school education and then we can link it to validity and reliability problems because usually these are the main things. So we can then get like. Or create a list of top tips of good assessment. And once they understand that we need face validity and construct validity, content Validity. We're not just like, for example, as Nate, you said, I want to test them, but I'm going to say something different than what appears on the test or I prepare them differently. And so they are going to be dissatisfied when they see the test. Test in person.
And these are super important elements of assessment literacy because of course, teachers can take a teacher's book and use a test from there, but they should be aware of these core concepts when they are assessing their students.
And they can assess their students without actually giving them a test. They can continuously assess while they are listening to them, and they are still using these concepts even without a test paper.
[00:06:19] Speaker C: Yeah. I mean, you mentioned reliability there very quickly.
[00:06:24] Speaker B: Oh, yeah. We need to define that as well.
[00:06:27] Speaker C: Well, yeah, without going into too much. But I'm thinking more about its. Its importance to a teacher. Why. Why is it important to understand reliability? I'm kind of looking across.
[00:06:37] Speaker A: Yeah. I mean, reliability is not necessarily something that many language teachers will. Will do or take into account. And reliability is actually a statistical property of a test. Right.
So not many teachers are going to sit down and calculate the reliability statistics of that test. But in a nutshell, what it means is if you gave the same test to the same group of students maybe two days apart, that you would get broadly comparable outcomes.
[00:07:00] Speaker C: Yeah.
[00:07:01] Speaker A: And so that is actually one kind of reliability that we talk about. That's. That's a kind of what we call a test retest reliability.
[00:07:06] Speaker C: And also, if a different teacher was evaluating you, you'd get the same result.
[00:07:10] Speaker A: Exactly.
[00:07:11] Speaker B: That's why standardization is an important thing.
[00:07:14] Speaker C: Yeah.
[00:07:14] Speaker B: That's another thing that I can standardize evaluators. And then we do that as well. That we.
And I think that should be done regularly, that testers standardize among themselves and they look at a speaking test, an essay or some kind of piece of work, and they standardize themselves and they figure out who's maybe too strict or too lenient. How, how we can pull this together.
[00:07:48] Speaker A: I think that's a question we should invite the audience to ask themselves. If, if instinctively you are against standardized testing, how. Just think about your own testing experience. If you didn't have any rubrics, if you didn't have a kind of. If you didn't have one test for everybody, if you gave a different test to everybody.
[00:08:07] Speaker C: Yeah.
[00:08:07] Speaker A: To what extent would your tests be meaningful?
To what extent would they be useful to you? So then just think about this notion of standardization and then think actually, well, standardization is in society. It's all around us. I mean, Even time is a form of standardization. How we count, weighing and.
[00:08:23] Speaker B: Yeah. Nowadays we tend to think that standardization is the worst thing ever because it keeps or puts people in boxes and sets everything. But yeah, we need to somehow keep ourselves up to a standard to see how we are measuring up or how we exist in society.
[00:08:43] Speaker C: I mean, I, you know, when you have large scale tests such as Oxford Test of English and you've got tens of thousands of people taking that test and you have to make sure that it's fair for everybody, the notion of reliability kind of seems obvious in that kind of situation. But you're a teacher in a classroom and you're assessing your students.
What, where could it go wrong? Because you're not thinking about reliability favoring certain students.
[00:09:12] Speaker B: For example. Yeah.
Also.
Yeah. Like for example, your personality and your emotional state can change the reliability of a test or, or evaluation as well. That, for example, today you're feeling a bit more stressed than the day before.
[00:09:33] Speaker C: Yeah.
[00:09:33] Speaker B: That shouldn't have any impact on the way you assess or evaluate. Yeah, yeah.
[00:09:40] Speaker C: And things, I mean, I know very much from my own experience as a teacher, you are influenced by how good a student is in class in terms of are they cooperative, do they do the things that you want them to do? Yeah. Okay. So I'm going to give them a better mark because I'm positively disposed to them.
[00:09:59] Speaker B: Also, if we think about this like testing, standardized testing measures your competence at a given point in time.
The problem when you're assessing your own students who you see every single day or every week, you. You know what they are capable of.
[00:10:16] Speaker C: Yes.
[00:10:16] Speaker B: So that might influence your assessment as well. That. Because you know that they can actually do a certain thing that shouldn't actually affect their evaluation in that certain moment or on that particular test, because that's that moment.
[00:10:35] Speaker C: I mean, I was going to say my, my experience, experience of the first time of going through examiner training, standardization was the shock. Realizing that other people who are as experienced as me just would give different scores to the same performance and realizing that we had just different assumptions about what qualifies as a good performance and talking that out. I think as a teacher that is.
[00:11:01] Speaker B: Really useful and super. Yes. Super useful. Like that needs to be talked about. I, I wish there was a time every term for teachers to come together, do some standardization, figure out how they are, how they assess the same performance. That. That's super educational.
[00:11:18] Speaker A: Yeah, yeah, yeah. I mean, we have to be pragmatic about these things and accept that what we're doing is social science and it is going to be. You know, the fact is, we do try to standardize to make everything fair, actually. I mean, we haven't really spoken about the relationship between standardization and fairness, but actually, I would argue that it is there. And that is. Fairness is actually the heart of why we have standardization.
I mean, of course, if you had a, you know, got somebody in here, gave them a tape measure, asked them to measure the table, then somebody else, yeah, same thing, Same table, same tape measure. Very, very similar results. But when you're measuring people and you're in a social situation, and when you're measuring something that you can't actually see, like language proficiency, suddenly that whole process becomes a lot more challenging and complicated.
[00:12:02] Speaker B: Can I. Can I mention an.
[00:12:03] Speaker A: Please, go ahead.
[00:12:05] Speaker B: I'm just thinking about. I've had one experience. I was doing presentation skills kind of preparation course again at university, and there was a student who, even at the beginning, it was obvious that she. She needs a lot of work and lots of improvement.
At the beginning, we did a diagnostic test. It turned out that she. She's.
Yeah, she needs a lot of work.
But then we worked. There was a lot of work put into this strategic improvement. And then at the end, we did another test.
She got better. Everyone was super helpful. We did peer feedback. We praised her a lot. Still, I had to tell her that you wouldn't pass the actual exam that we are preparing for. So I'm very happy about your progress. You made a lot of progress throughout the course, but you're not there yet. And that is all about fairness and standardization and everything. I know I could feel I could see there was visible improvement, but it was still not enough. And as a teacher, you have that responsibility to be open and fair in this respect as well. That you have to acknowledge that. Okay, a lot of things have happened, but you're not there yet.
[00:13:33] Speaker C: That's such a good point. I was also thinking of what you mentioned earlier about asking students, teacher trainees, about their own experience and what they didn't like. And I'm sure that a number of them would say that their experience of assessment was lack of fairness because some teachers would give different results to others.
And that's. You feel so embittered as a student, you know that some teachers are more generous than others, and the only way to deal with that is through standardization.
Yeah.
[00:14:08] Speaker A: We have to accept that failing a test is not necessarily unjust and communicate that in such a way, you know, that the students can understand that. And in actual fact, you know, maybe actually falsely passing someone is potentially worse because the consequences of that are potentially.
[00:14:27] Speaker B: Greater because you give them that confidence.
[00:14:31] Speaker A: Certify someone for something they cannot do.
[00:14:33] Speaker B: Yes.
[00:14:34] Speaker A: Well, then you're simply delaying what might happen later on in terms of a failure.
[00:14:39] Speaker B: And you mentioned the machine operation. You mentioned driving tests. I mean, you wouldn't certify someone who cannot operate a machine well or who cannot drive safely because that would lead to terrible consequences. And, yeah, it's just the same.
[00:14:55] Speaker A: My father was a driving examiner, and I suspect that that's where it comes. Where it comes from to a certain extent. Maybe there is some latent influence there as well.
And I've always thought that the impact and the stakes of a driving test are much higher than the stakes of a language test, to be honest.
[00:15:11] Speaker C: That's true.
[00:15:12] Speaker A: Certifying people to drive on the road versus certifying someone to use language in a university. Yes, I think. Yeah, but one is life and death and the other is not at the end of the day.
[00:15:23] Speaker B: And also, I think if you communicate everything fairly and you communicate your evaluation fairly, I mean, I failed my first driving test, but I remember why I wasn't ready yet, and there were so many things happening. I got got scared and stressed and. And that shows that you're actually not fit for that at that point. So, of course I was sad, but I could understand and I could accept that, okay, this actually shows that I'm not there yet. If we can communicate failure in this manner, then they can also feel that, okay, maybe it's. It's just. It's right, and I just need to try again.
[00:16:04] Speaker A: It's much more preferable that we say to that student, okay, you're not quite there yet, but if you take your language test in a few weeks, a few months, or maybe another year, in some cases, you will be better prepared for whatever it is that's coming next. That is surely a better outcome than falsely passing a language test, then going to higher education, and then really, really struggling in your course because you can't participate in the discussion. Dropping out, in fact, and then dropping out.
[00:16:28] Speaker B: Yeah. Hi, I'm Joe Sokey. If you want to discover more about how to use tests and assessment as tools for growth in the English language classroom and the concept of positive washback in more detail. Download our position paper that I co authored alongside other amazing contributors called the Impact of Assessment on Teaching and Creative Positive Washback. In this paper, you can explore research on how testing and assessment shape classroom practices and find practical guidance and advice on how you can prepare your students for exams while also addressing their broader language needs. Download the paper via the link in the description and enjoy the read.
[00:17:07] Speaker A: Thanks.
[00:17:08] Speaker C: I think a shift to the side and think of this something you mentioned earlier around the negative washback and you find yourself as a teacher in a situation where you have an. You feel you understand what the students need, but they're also preparing for an exam which doesn't focus particularly on the things that you think they need in terms of communication skills.
You know, so, so you got an exam. Maybe it's very grammar, vocabulary, multiple choice focused. They, you know, you have to get them through that, but you also want to develop their actual communication skills.
Have you got any advice for those poor teachers out there who are struggling with this dilemma? Do they teach to the test?
How do they manage that situation?
[00:17:58] Speaker B: If I can do this, please start first.
I think yet this is a super difficult question because they know they have to achieve something. Maybe the parents are there, maybe just the student is an adult student and they have their high expectations.
Honestly, if I were them, these teachers, I would be honest and open and transparent about this that, listen, I know that we need to pass this test, but I also want to improve your communication skills. This test might not actually assess your communication skills or put a huge emphasis on it. So we're going to divide the lesson up into two halves. 1/2 is for preparing for the test, the other is preparing for life. Yeah, if the two are so different, but if I can somehow consolidate the two and still make it work, that would be the best because then it's not such a clear division between the two. But I, I'm aware of tests that are super grammar based and they don't put emphasis on speaking or communication.
[00:19:08] Speaker A: Yeah, yeah, yeah. Like that's more or less what I was going to say. To be honest. That's a very good way of putting it.
All I would add really is yes, I think if you're confronted with something like just a pure grammar test, invariably it's going to be multiple choice. But the way multiple choice questions are designed is that they're very often like little excerpts of speech. So it might be like two people in a shop or something and you've got one which is a question and which is the correct response. So I would advise to a teacher, let's have a look at this kind of bank of questions that you've got as kind of preparatory materials. See if you can't group them together in some way thematically and they say, okay, we'll try and get into the Head of the item writers a little bit.
The people who created the test because they're thinking about scenarios and language use, even if it's not explicit.
[00:20:01] Speaker B: Oh, if the question is like that. I really like these types of questions because they actually. Yes, because they try to bring in communication and scenarios.
The ones that just want you to find one grammar item that's like yeah.
[00:20:16] Speaker C: Yeah, yeah, yeah, yeah. Okay, that's really interesting. We're talking about how to combine genuine communicative language learning with preparing for the test.
Maybe we don't always have to be so explicit about it. Are there ways of preparing for the exam which don't sound like it's preparing for the exam?
[00:20:40] Speaker B: Yeah, I think it's sometimes useful because it can be definitely useful to be explicit and tell them directly that we're doing this for a reason and that reason is preparing for the test. But sometimes it can be counteractive and debilitating. So it could be a good idea to complete an activity which is a communicative activity and then unpack once we're done. Okay, so what kind of skills did we use in this test activity and how those skills connect to the exam?
And this can maybe help students become more confident, act as if they were normal, like in a normal situation.
Take away all that anxiety, test anxiety and then.
And actually we can, I guess, get there, buy in as well. So they might say that this is not on the exam or we only want to do things that are on the exam. But with this we can make them complete certain communicative tasks and then work it out with, together with them how it connected actually to the test.
[00:21:54] Speaker C: Yeah, I mean, is there a particular example you might give of trying to.
[00:22:00] Speaker B: For example, like I recently, not sort of recently did a kind of debate activity. So it was.
They had to.
It's a very simple task. It's a kind of a pyramid speaking task. They had to come up with ideas in pairs. Then the pairs opened up to fours and then the fours became a big eight, group of eight. They had to keep convincing one another of how or why they think something should go in onto their, onto the list.
This can go on forever. Practically.
They might think that, oh, this is exciting but not particularly useful. However, at the end when you tell them that, okay, can we just look at the skills that you use during this, sorry, this task. They can then unpack that. Okay, we use negotiation skills. We did turn taking. We tried to convince each other. We, we did a bit of agreeing, disagreeing and then we. Maybe we can then collect the the phrases they used for these particular sub skills, then we can say, okay, so these are all the things you can use in a discussion task on an exam.
Individual turn, for example.
And, and that's actually a very, very good task. Low prep and easily gives you like 60 minutes.
[00:23:24] Speaker C: That's a gem.
That's what we want.
[00:23:26] Speaker A: More of those.
[00:23:27] Speaker C: Yeah. Except it needs quite a lot of skill on the side of the teachers. Yes, yeah. To respond.
[00:23:32] Speaker B: That's why, that's why actually like these dog me style teaching activities and are very teacher like, not teacher heavy, but they need a lot of skills from the teacher.
[00:23:45] Speaker A: Yeah, yeah. I think if you are clarifying what skills are involved in any classroom activity, whether you're doing that beforehand or whether you're doing it post hoc, both equally valuable.
But yeah, as long as that kind of clarification is there for the learners about why they're doing something and how it relates to it, even if it's indirectly, then I think they can see the value of it. Absolutely.
[00:24:05] Speaker C: It's interesting you say that because we. In a previous podcast series about Gen Alpha, one of the points that was being made was that that Gen Alpha brought up in a much more responsive world where things happen kind of quickly in response to what you asked for. And that kind of makes them impatient to do things which it's not clear what the purpose is and that's probably feeding into this.
But, sir, is it in the test, you know, kind of response, you know?
[00:24:35] Speaker B: Yeah.
[00:24:36] Speaker C: Which is tough for teachers, isn't it?
[00:24:38] Speaker B: Yeah. You can, you can make the task shorter and then, or, or. I mean, I, I completely can sympathize with Jan Alpha on this because I'm super impatient and, and I feel like, okay, I want to do things that make sense and I don't want my time to be wasted. But then you can just cut it up into smaller sections that, okay, we're just gonna do 10 minutes of this larger activity and then stop after 10 minutes and then reflect.
In that case, you kind of give away what you had in mind, but at least with these smaller chunks you can keep them engaged.
[00:25:18] Speaker C: I think we've really explored this topic about what it means for teachers in the classroom.
Thank you so much. Thank you, thank you, thank you.
