Episode Transcript
[00:00:10] Speaker A: Hey everyone and welcome to Talking ELT, the easiest place to learn about the big trends in language teaching. Today. We're continuing our conversation about artificial intelligence with higher reindeers and Ben Knight. Welcome back, guys. It's so we've been talking a lot already about the impact on teaching and we've touched a couple of times in that conversation on feedback. And I think I would like to move us on to broadly what the impact on assessment is going to be because I think we've kept coming back to this because the two are intrinsically linked. So yes, I think that's probably our next topic. So more broadly, zooming out. What do you think the impact of AI on assessment might be?
[00:00:55] Speaker B: I think people are already aware that the use of certain assessment techniques are not going to work in the future. So just asking people to write an assignment was nice and easy way of assessing someone's language skills. That is not the future unless it's.
[00:01:15] Speaker A: Done by hand in the classroom, potentially.
[00:01:17] Speaker B: Right, okay, yeah, you can do that. And you can kind of get people to scribe with a quill or something, whether that's going to help them in the future. No, probably not.
So I think in that sense, assessment is going to have to adapt to what are the skills needed in the workplace or whatever life engagements you have.
And since that is more likely going to be the creative use of generative AI, the effective use of it, I imagine that's going to be a key part of assessment in the future.
[00:01:55] Speaker C: I would take that one step further and I would say that maybe not in the next year or two, but in the not too distant future, assessment is no longer a separate term. And the reason why I say that is because the computational turn, which is the emergence of a whole range of different opportunities afforded by things like big data, as well as the ability to use analytics to extract information. From it and so on and so forth, actually means that possibly already now, but certainly in the not too distant future, you'll be able to essentially see as a teacher, as a school, as an administrator, or even as an individual, learner, by the way, in real time everything that is happening, all the interactions. So let's make it more concrete. Let's say that you're a teacher. You've got 20 learners in your class. You'll be able to see what everyone is interacting with because there'll be some digital trace, there'll be some record, because the majority of what learners are already doing, but certainly will be doing, will involve or at least be mediated by technology in some way. So you can literally almost like watching the water and a river flow past. You can just go and fish in it at a moment in time and say, this is the level of my load. This is what they're engaging with. This is what they seem to be interested in. And so assessment becomes, in a way, almost meaningless. And I think the most important reason for that is because as an individual learner.
And we're not quite there yet. And in fact, I'm developing something myself at the moment around this.
Maybe at another podcast we can talk. About this is that as an individual learner, you should be able to capture that at any moment in time and then present it to either a teacher or an employer or a colleague, et cetera. And that can then give that person, of course, based on your personal choice, what you want to share or what you don't want to share. That person an imaginably comprehensive and detailed insight into not just a few test scores that you observe, but everything that you've done in a particular domain. Right, yeah. Right. That's fascinating.
[00:04:23] Speaker B: I always want to make this distinction between formative assessment and continuous summative assessment, of course, so it may be that that disappears, but at the moment, people might do weekly tests or unit tests and those might be recorded, and those are built up to an end of year evaluation.
And that's what I would call continuous summative.
It's going to be used in some way to evaluate their success.
But a formative assessment, obviously, as you know, is more about information going back to the student.
And there are times when the student wants to be confident that that's not going to be part of their summative assessment, so that they can explore, they can try things out and not worry about whether it's going to affect their end of year grade.
At the moment, I feel we have to kind of keep those sets of data almost separate. And perhaps that was what you were saying when you're saying that the learner might be able to choose what data they're sharing. So you're kind of distinguishing data, which is about your performance in practice, kind of exploratory activities and activities which are more a check on what you can actually do. But it may be that we actually get to a stage where there's so much data that each individual activity has.
That distinction is important because each activity has such a small impact on the final evaluation exactly. That it doesn't matter anymore.
[00:06:05] Speaker C: Yeah. And I guess it brings us right back down to the underlying question what is the purpose of different types of assessment? Well, broadly speaking, it's either an identification of what a person can do and what they know and what they believe, perhaps in a certain domain, which is summative, perhaps. And then formative is providing information to the learner to enhance their knowledge and skills, et cetera, in a particular domain. Right, yeah. That distinction, I think, is completely gone because as you said, if I make one mistake in class today, that's not going to have a major impact on my profile or my body of knowledge and skills in a particular domain. Right? Yeah. And so as an employer, for example, or as an admissions officer well, here's where it gets really interesting, because let's say that you give me your profile and maybe you've made a pre selection. Maybe you haven't included, if you're applying for postgraduate degree here at Oxford University in English Literature, maybe when you submit your application, you don't include your math exam scores or your math performance. Right. I certainly wouldn't. I can tell you that. Right. So you may have curated, as a learner, interesting, some of what you wish to share. Now, as an admissions officer or an employer or whatever the case may be, I can then engage possibly even in conversation or using natural language processing with that body of knowledge.
[00:07:39] Speaker B: Right?
[00:07:41] Speaker C: And maybe I do it as a human being, or maybe I use my own curated AI. It doesn't matter. But then I can query for a particular context, like a particular job or a particular program. I can say this program consists of the following elements. It requires the following attitudes and skills and knowledge, et cetera. To what extent does Ben have the ability to engage in X, Y and Z in the context of this particular situation, for example? And then I get a level of insight into your aptitude and your potential for whatever that may be, that is infinitely more useful than your exam scores.
[00:08:27] Speaker A: Yeah, of course.
[00:08:27] Speaker B: Absolutely.
At the moment when we talk about assessment, we're often distinguishing between scores and kind of portfolios of evidence. Those are two different things. But they don't have to be. Portfolios of evidence are essentially data which can be interrogated, as you say. So, again, a kind of nice blurring of something that's a problem at the moment. That distinction.
[00:08:55] Speaker C: Again, unanticipated consequences, because that leads to a lot of questions and a lot of potential dangers. But I also see a lot of potential opportunities because going back to the earlier example, if as a learner, as an individual, doesn't even have to be just in education, but if I now have a level of control over, for example, what I wish to share, right? And if I can actually show you all of the things that I've done, then that is potentially a lot more beneficial to me than just an individual score, which was derived from a time when perhaps I had a headache and didn't do my best. Or I might be able to share with you the things that I did outside of formal education, which are not captured normally through assessments, et cetera, but which may be equally important for the job that I'm applying for.
[00:09:56] Speaker B: I think that's such a good point because, as we know, language competence is multidimensional.
We all kind of object to the way that it's summed up as a single score or a single grade. But the need for that is that human beings are making decisions about whether you're suitable for something or other and they can only handle that one dimension. And that's why it gets summed up as a single score. If those decisions are being aided by an AI system which is able to interrogate to handle more than one dimension of your competence, then there's no need to reduce everything to single scores.
[00:10:41] Speaker C: And even if you do have a sophisticated system that allows for a multivariate analysis of different variables that may impact on your global score or what have you, that is only a measure of your constellation of abilities, et cetera, in one moment in time. And so if we look at something like complex dynamic systems theory, which of course in the last few years has really started to become more recognized for its potential in our field sorry, can.
[00:11:14] Speaker A: I just clarify what that is?
[00:11:16] Speaker C: I was just about to tell you then it allows for a recognition of all of the different variables that not only impact directly on what you're interested in observing, for example, your language proficiency scores, but also shows how all of these different variables impact each other in what direction. So for example, if you take a complex construct like motivation, which we talked about yesterday, then you can see that something might impact motivation, but not directly, it might be through something else.
And if you're able to capture that complexity and if you're able to capture that not just at one moment in time, but if you're able to do that over a period of time and see how it shifts and turns and what have you as a result, perhaps, of your instruction or the feedback that I've received or the materials that I've engaged with in addition to the things that I do outside of the classroom. In addition to the social interaction that I have with my classroom peers, well, that is a far more rich picture than a single measure would be able to give us.
[00:12:25] Speaker B: What do you think of a lot of the kind of Gamification that is kind of being built into learning systems?
And it sometimes has a feel to me of almost behaviorist approach. You get rewarded or you get punished, depending on whether you get something right or not.
When we talk about delivery, I suppose this links onto feedback as well.
Is that the direction that we're going in? Is it going to become more and more like that?
[00:12:58] Speaker C: It's a really interesting question, actually, because you mentioned it's related to feedback and I guess there's a blurring which didn't exist before between, what do you call it, an immediate recast that a teacher might use a correction, right? You make a mistake and I correct it in the context of a conversation or what have you, separate from say, an exam or a test.
And this almost kind of blurs the boundaries between those two. But to answer your question with the Gamification, especially the behavioral or behavioristic approach to that I'm actually increasingly ambivalent about that because for a long time, and I think that will be the general atmosphere in our field is that, well, we've moved away from behaviorism. So surely this is something from the 1950s and we know it doesn't work. But then you look at, for example, metaanalyses like by John Hattie, and you find that checklists are actually I think it might even be the greatest impact of all the different feedback measures on learners performance.
Having a simple checklist. Did you do this? Did you make sure? Did you compare? Did you check?
So again, it's that question of efficiency and effectiveness, because maybe a language lab from the 1950s or doing checklists all day every day, or your example of the personalized university could be probably possibly the most efficient way of doing things.
[00:14:35] Speaker B: Yeah, no, I agree with you.
[00:14:37] Speaker C: Not the most effective, especially not if learners are going to be bored and become demotivated and disengaged from the learning process entirely.
[00:14:46] Speaker B: Yeah, no, I think that's really interesting.
[00:14:48] Speaker A: Thanks for listening to this episode of Talking ELT, the easiest place to learn about the big issues in language teaching. Don't forget to like and subscribe if.
[00:14:59] Speaker C: You want to learn more about this.
[00:15:00] Speaker A: Issue and others like it. We look forward to seeing you next episode.
