Episode Transcript
[00:00:11] Speaker A: Hi, everyone, and welcome back to talking.
[00:00:13] Speaker B: ELT, the easiest place to learn about the big issues in language teaching. Last episode, we talked about how we can help our learners both critically view and express themselves through multimedia, such as images, video and audio.
We're joined again today by Ed, Charlotte and Nick, and we'll be continuing our conversation on multimodality.
[00:00:35] Speaker A: So I'm going to move us on to another big issue, which is assessment. And we touched on this previously, but I think it's something that our listeners are probably going to be quite interested in because we talked a lot about how do you create, how do you help learners create multimodal texts? How do you help them understand and interact with multimodal texts? And I'm sure everyone's wondering, well, how do we assess this? How do we. How do we make sure that they're actually achieving this and gaining these skills?
So, yeah, my first question is, how can we go about assessing multimodal literacy?
[00:01:10] Speaker C: I mean, I think that's an interesting one and something I've gone very broad happening at a very interesting time.
[00:01:17] Speaker A: Okay.
[00:01:17] Speaker C: Because, you know, I think at the moment, with, with what's happening in AI, assessment is in crisis. Really the whole assessment. We're coming to the point where any student can just sort of put any question into chat GPT and get the answer produced for them. They'll be able to do that with audio and video as well. And I think this is where representation really comes into its strength, because if we are designing assessments that students do through personal representation, it's very hard, least at the moment, to take a video of yourself.
[00:01:52] Speaker D: Yeah.
[00:01:53] Speaker C: You know?
[00:01:53] Speaker D: Yeah.
[00:01:53] Speaker C: So if you're creating something and you're doing something creative, you know, then that becomes your. And that becomes your form of assessment. It's very difficult to get AI to do that for you, I think.
[00:02:06] Speaker D: Yeah. Yeah.
[00:02:07] Speaker C: So, I mean, that's kind of twisted, your, your question, in a way, because you still have the problem of, well, how do you assess that? But, you know, I think it solves one problem. You know, we still need sort of criteria around what. How to evaluate what students produce. Yeah, but, you know, that isn't such a huge step, I think, to sort of, you know, cover that.
[00:02:31] Speaker E: No, I think the, the criteria that we have for the four skills, the, the traditional kind of means of communicating are a helpful starting point for that. And as long as the, the.
The input and the output that we're using to assess is kind of mirrored by what we've been using in the classroom to teach and to help learning, then there's a chance there to build an assessment program, which is a proper culmination of the teaching that we've been doing. But I think students have to be really clear about what the assessment criteria are and what their work, whether it's some kind of continuous assessment or portfolio assessment, what it needs to contain, what the criteria are that it needs to be, to be, to be hitting in order for it to work. My fear, I suppose, is that that's quite a big task to do. It's quite difficult for a school to say, well, listen, we've been operating really well, preparing students for university entrance or whatever it is.
We don't really feel like changing that right now. Long story short, if we're going to test it, we should be teaching it as well. I think that just underlines what we've been talking about so far today.
[00:03:39] Speaker A: But then I suppose at the same time, if we're going to teach it, we have to test it. It's one of those ones where so much of teaching is guided by assessment practices that in order to actually teach effectively, we need to shift the way that schools and curriculums and ministries think about assessment, I guess.
[00:03:57] Speaker E: Yeah, absolutely.
[00:03:58] Speaker C: And it is very difficult to do. I've been trying to write a multimedia syllabus and trying to level different things from multi. Why couldn't we have come up with an easier multimodal literacy kind of syllabus of, you know, what then becomes b one body language as opposed to c, two body language?
[00:04:21] Speaker A: Okay.
[00:04:21] Speaker D: Yeah.
[00:04:22] Speaker C: You know, how do you fit that into your syllabus, your curriculum? It's very difficult to level those things, you know? And also in terms of language teachers, you know, they're very complex things to talk about.
[00:04:34] Speaker D: Yes.
[00:04:35] Speaker C: So how do you deal with talking about them with your students if you're in a multilingual cast, for example? Yeah, you know, how do you, you know, if, for example, I was talking earlier about, you know, breaking film down into different shots and which different shots you use, you know, how do you level that and how do you talk, how do you talk about those things if you're in very basic English, you know, that becomes very difficult.
[00:05:02] Speaker F: I think a big red flag for me is that it doesn't relate to anything that our students are asked to do when we're thinking about those big exams.
And therefore it leads to the question of, well, why would I prepare my learner to record an audio clip as part of an assessment if they're not actually going to be assessed on it for the purpose of them getting the grade they need to progress to university, for example.
So it kind of ties into what you were saying, chris, about, you know, there almost needs to be a shift higher up to emphasize maybe the importance of assessment that involves some form of multimodal literacies.
[00:06:01] Speaker C: Is the shift not happening, though, because the shift is happening in the workplace, isn't it? You know, what workplaces need is becoming very different, and what universities require is going to change as well, I think.
[00:06:13] Speaker F: Great question.
[00:06:14] Speaker C: Maybe that's happened.
[00:06:15] Speaker F: Yeah, I mean, I mean, I don't know. I mean, I can't speak for, you know, ministries of education around the world and whether this is something that they're starting to pay more attention to. But I'd like to hope it is because, as you say, it's gearing learners up for that world of work once they're done.
[00:06:34] Speaker E: That's the incentive, isn't it? That's the incentive. Like, in your workplace, you will have to give presentations. Those presentations will have to contain multimedia, so you'll be giving a presentation as part of your assessment. And I think this links back to something you said earlier, Charlotte, about Clil, where we have to take the focus off the language for a certain part of the assessment and to say we're actually looking both at your ability to use language as a student to convey a message and also your ability to marshal various aspects of multimedia to make that message more effective. And I guess it's also something where in the assessment, we also need to contrive an audience. Right. Because we've been talking about how important it is to have someone to, to receive or to interact with the message as it comes through multimedia. Much harder to do that in an assessment situation, unless it's a presentation or something which is posted or something which is accessible to people other than the examiner. But then I guess that kind of brings with it a whole other bunch of potential difficulties of privacy and confidentiality and so on.
[00:07:40] Speaker A: Of course.
[00:07:40] Speaker D: Yeah.
[00:07:42] Speaker F: And I think also it's that criteria that Nick mentioned earlier as well. You know, if you're giving your learners some kind of, let's say, a really basic, you know, test of, I don't know, a gap fill. And so it's very clear what, you know, what the right and the wrong answers are. But does it become more subjective when we're getting our learners to create some kind of media piece that they're going to be assessed on? What is it that we're actually assessing them on, and how do we create the criteria around that assessment? How do we create grading around that.
Is it as objective as the gap fill exercise? And that's quite a challenge, I think.
And I would question whether there is much out there that supports assessment in that way.
[00:08:40] Speaker E: I guess it's all emerging and emerging.
[00:08:42] Speaker C: Fast, getting marked down because you used the wrong font or something like that.
You used comic sans, I'm afraid.
[00:08:51] Speaker D: Yeah.
[00:08:51] Speaker A: I mean, you get zero if you use comic sans. But I do think that is an interesting point because I think there is a tendency and a risk of assessing the things we've always assessed. So, for example, if you're showing someone a video and you want them to do a comprehension task, there's a risk that you're just going to assess their ability to comprehend the language that's spoken, where 90% of the marks will just be speaking. And then at that point, it is just a listening assessment. It's not actually assessing their ability to comprehend the video as a whole.
So I think we need to leave room. Whatever rubrics we come up with to define it, we need to leave room to assess. Okay, what role does music play in this piece, and how have you commented on that? The visuals and what the characters are doing on screen and body language and all of that. How do you assess those elements?
How have you commented on those elements? I suppose it's about putting as much weight on those features as it is on the traditional language skills.
[00:09:59] Speaker C: There's also a resource issue there as well, not just in terms of teachers, but, you know, if you're thinking about a natural assessment of your students when they're 18 and having to evaluate thousands and thousands of videos, you know, is it something we can palm off onto AI to do yet?
[00:10:17] Speaker D: Yeah, probably not.
[00:10:17] Speaker C: Probably not.
[00:10:18] Speaker D: Probably not.
[00:10:19] Speaker F: Yeah. Where's the automation in it?
[00:10:22] Speaker D: Yeah.
[00:10:22] Speaker C: Because, you know, an awful lot of assessment is done the way it is because it's really easy to do. You know, it's really easy to get a computer to see who's got the right number of answers on a multiple choice question.
[00:10:34] Speaker A: Yeah.
[00:10:36] Speaker C: When you start evaluating soft skills and things like that, that becomes much, much more difficult to do, much more time consuming.
[00:10:44] Speaker E: I was struck, Nick, by what you said earlier about the arrival of AI and how that's really put assessment in danger. And I was hearing about certain schools who are just requiring students to come in and do more exams in exam rooms to do more things where there's this kind of proof that it's not been done using the help of AI. And maybe that is the great enabling factor for multimodal literacies that through doing something which is really difficult to replicate using machine intelligence, we can actually have something which is almost unfalsifiable. Do you know what I mean?
This is the reason we're using multimodal elements in our assessment, because it's cheat. We feel it's cheat proof in a way that we want you to record yourselves. We want to see your face and hear your voice, we want to ask you questions which you can't use. You can't easily use AI to get the answer to because it's personal and it's based on your opinion and it's open ended and it's slightly unpredictable. So that's a really interesting point you made.
[00:11:53] Speaker C: I'll throw something the other way now in that know, shouldn't we also be teaching our students how to use AI to produce these things as well? Because that's, you know, an element of the reality of their lives in the future as well, isn't it? Maybe that's a discussion for another time.
[00:12:09] Speaker A: No, but it's always, you know, it's.
[00:12:11] Speaker C: Certainly going to become something that's a fact of life. You know, they're going to be using AI to help them create video, to create audio, to create text.
[00:12:21] Speaker D: All of them.
[00:12:22] Speaker F: Yeah, I mean, I was on the train the other week and I overheard a girl saying how she needed to do her cv because she wanted to apply for a job and a stranger just interrupted and said, oh, you should use chat GPT for that because. And she'd not heard of it, which surprised me because I feel like, you know, it's quite a sort of, you know, topic of conversation nowadays. But maybe that's just because we're working education and we're more aware of it than, than other people. But, you know, this, this guy on the train, you know, he could have got commission for chat GPT, you know, he was really promoting it as this really useful tool for helping, you know, you with a seemingly quite boring task, updating your cv, you know, but I think there's also the element of, you know, maybe chat GPT knows the best way to create a cv, so it's going to stay stand out more against the noise of all the other cv's which come through for a drop for.
[00:13:20] Speaker E: A certain period of time anyway, until everyone's using it.
[00:13:22] Speaker F: Until everyone's using it.
[00:13:24] Speaker C: Interesting timing. Over lunch I got an email from someone. I've been asked to write two blog posts for a company and over lunch they sent me a follow up email and saying, if you use AI or chat GPT to do this.
Can you let us know which parts of it were it's okay for you to use it, but can you let us know which parts you've used it for so that we know which parts are authentic and things like that? Which was really interesting. It was really interesting.
I thought so anyway.
[00:13:55] Speaker D: No, no.
[00:13:55] Speaker A: That's becoming a thing which people are needing to take into account to such an extent and becoming a thing that.
[00:14:02] Speaker F: People are acknowledging that people will use.
[00:14:05] Speaker D: Yes.
[00:14:06] Speaker A: Yeah, it feels like we've been, we've been talking a lot about assessing students productive skills and their ability to create multimodal text, and that's been a lot of the problems we've come up against.
What about helping?
[00:14:19] Speaker F: What about.
[00:14:19] Speaker A: What about assessing their receptive skills so their viewing skills, their ability to engage with and critically understand and analyze multimodal texts? How can we make sure we're doing that in a way that is accounting for these multimodal skills?
[00:14:32] Speaker E: I mean, obviously it builds on the receptive skills of listening and reading. So those go without saying that they need to be taught, coached and practiced.
I'm thinking that some of the prompts that were often used for productive skills, like a picture description, for example, could be used for receptive skills as well. So what do you really see going on here? Where it could be a combination of images which are ambiguous, or a combination of images which are slightly manipulative, perhaps, and to give students a chance to demonstrate their ability to look at or receive or respond to multimedia in various, from various angles. So I think that using the traditional kind of prompt that we have is not going to be enough. But actually deliberately finding images, a combination of text types, multimedia, and using those as prompts for receptive skills. What did you see here? What did you hear here? What was going on here? And giving students a chance to demonstrate those sensibilities, or that understanding would be definitely one way forward.
I was thinking of an example, a text that you might see, a short text that might be, say, it might be ten words, like a headline or something like that. We would often ask students to kind of interpret a headline. But what about if that headline is seen in three different places? One could be like, on a newspaper. One could be a tattoo on someone's body. A third one could be like, on the projected on the side of a building, a world famous building. Like, what's the difference in similarity between those three versions of the same text? That's a more interesting exploration of something which is text but also much more than text.
[00:16:32] Speaker D: Yeah.
[00:16:32] Speaker A: And then it's making them look beyond the text at the other elements, I think. Yeah, that's so much of it. Looking at assessing, making sure the assessment is focused on the other areas as well.
[00:16:41] Speaker D: Yeah.
[00:16:44] Speaker C: Yeah. I mean, I think it's easier to do the receptive side side of it than it is to do the productive side in a way, because there is, you know, there is, if you like, in a way, a correct answer or something closer to being the correct answer, isn't there?
[00:16:58] Speaker D: Yeah.
[00:16:58] Speaker C: That you can define. You can sort of get a video, a piece of video, show it to students without the sound, get them to predict what they're talking about and, you know, you can say, okay, well, you got it right or you got it totally wrong or maybe, you know, they're somewhere in between.
[00:17:13] Speaker D: Yeah.
[00:17:13] Speaker C: You had something logical, but. Yeah, and. But I think so. I think to some extent that's a bit easier than the productive side.
[00:17:23] Speaker D: Yeah, I would agree with that.
[00:17:24] Speaker C: As long as we are asking the questions, are we asking, you know, the right questions?
[00:17:29] Speaker D: Yeah.
[00:17:29] Speaker C: Becoming part of the assessment, you know, and we're not just focusing on, you know, listening and understanding the words and.
[00:17:36] Speaker A: We'Re focusing on those areas which we also need to be able to. Assessing.
[00:17:39] Speaker E: Assessment was hard.
[00:17:41] Speaker A: Assessment was hard.
[00:17:42] Speaker F: Assessment is always hard because we don't really know the answer, do we?
[00:17:45] Speaker D: Yeah.
[00:17:46] Speaker F: And it feels very emergent.
[00:17:51] Speaker D: Yeah.
[00:17:51] Speaker C: It's also something very difficult to change as well, isn't it? You know, changing assessment takes a long time.
It's a complicated thing and there's a lot of money behind, you know, keeping it this way.
[00:18:04] Speaker D: It is.
[00:18:05] Speaker C: And. And it's expensive to change as well, isn't it?
[00:18:07] Speaker F: Yeah.
We've recently explored frameworks for assessment and, you know, to try and create your own framework is such a mammoth task. It involves a lot of research and a lot of testing, you know, to see that it's valid and that's before.
[00:18:27] Speaker C: You try and get people to accept it as well.
On a kind of global level, it's very difficult.
[00:18:34] Speaker F: Yeah. And everything needs to be endorsed. Nowadays, if you don't have that endorsement, then no one's going to invest in it. So it's. Yeah. Getting the buy in from the right people.
[00:18:46] Speaker D: Yeah.
[00:18:46] Speaker F: And why would people change, you know, so you've got to convince them why it's worthwhile to make that upheaval, to move on to something that's evolved from where they were before.
[00:19:00] Speaker D: Yeah.
[00:19:00] Speaker A: But I suppose when you do have the need out there in the world, like we've talked about, companies needing people who are social media literate, people who are able to create video and create all this content and deliver presentations. I suppose that when those skills are in demand, that does create a driver.
[00:19:17] Speaker C: I'd like to think that's true.
But, you know, you look at the education system that we have at the moment and it doesn't prepare students, kids for the reality of a working life.
They're taught maths in a very abstract way, but not taught financial literacy. Who left school knowing how to fill in a tax return. Exactly.
Or get funding for their startup company.
[00:19:41] Speaker F: Apply for a mortgage, which all life skills.
I think the change needs to trickle in. I think it's not something that's just going to happen to overnight, of course, but I think, you know, you can trickle it in or just sort of, you know, make a few tweaks here and there. But it does mean that it's going to be a long process.
[00:20:06] Speaker D: Yeah.
[00:20:07] Speaker A: And it's going to take time. It's not happening overnight.
[00:20:09] Speaker C: Yeah.
[00:20:10] Speaker E: I mean, I'm right in thinking, aren't I, that they do actually use multimodal input receptively in some language exams. Right. Like you're asked to look at a map or two versions of the same.
[00:20:22] Speaker C: Map in ielts, things like graphs.
[00:20:25] Speaker E: Yeah.
[00:20:26] Speaker C: Things like that for the academic side.
[00:20:28] Speaker E: And I think, like, history papers have always used like, propaganda posters and things like that, as it's not, it's not that much of a brand new domain.
[00:20:38] Speaker A: No, no, no, I think you're right. I think you're right. I think a lot of it is, um.
I think a lot of it is happening gradually.
And I think potentially part of the shift is looking at it as an explicit skill that needs to be assessed in its own right. But those elements are in many ways already there. I think that is a really good point.
[00:21:03] Speaker E: Yeah. I'm just always uncertain as to. As to what really is happening with all things assessment. It's just, it's a specialized area.
[00:21:15] Speaker D: Yeah, yeah.
[00:21:17] Speaker A: Really interesting.
[00:21:19] Speaker C: I mean, if you think about university assessment, it hasn't changed for centuries. Exactly.
[00:21:24] Speaker D: No, no.
[00:21:26] Speaker C: Is it gonna change now? It is great that AI has happened, really, because it shakes things.
[00:21:31] Speaker A: Shakes everything up.
[00:21:32] Speaker D: Yeah.
[00:21:32] Speaker C: You know, then you have to make a change.
[00:21:34] Speaker F: Yeah. I mean, COVID should come everything up as well, didn't it?
[00:21:38] Speaker D: It did, yeah.
[00:21:39] Speaker F: So sometimes it does take these sort of. Yeah. External forces to create this change and.
[00:21:46] Speaker A: It requires you to.
[00:21:47] Speaker D: Yeah.
[00:21:47] Speaker B: Thanks for listening to this episode of talking ELT, the easiest place to learn about the big issues in language teaching.
[00:21:55] Speaker A: Don't forget to like and subscribe if.
[00:21:57] Speaker B: You want to learn more about this issue and others like it.
[00:22:01] Speaker A: You can also get practical resources and more advice on this topic by downloading our position paper. Just follow the link in the description.
