The two sigma problem

0:04 So anyone who's been paying attention for the last few months has been seeing headlines like this, especially in education. The thesis has been: students are going to be using ChatGPT and other forms of AI to cheat, do their assignments. They’re not going to learn. And it’s going to completely undermine education as we know it. Now, what I'm going to argue today is not only are there ways to mitigate all of that, if we put the right guardrails, we do the right things, we can mitigate it.

0:27 But I think we're at the cusp of using AI for probably the biggest positive transformation that education has ever seen. And the way we're going to do that is by giving every student on the planet an artificially intelligent but amazing personal tutor. And we're going to give every teacher on the planet an amazing, artificially intelligent teaching assistant. And just to appreciate how big of a deal it would be to give everyone a personal tutor, I show you this clip from Benjamin Bloom’s 1984 2 sigma study, or he called it the “2 sigma problem.” The 2 sigma comes from two standard deviation, sigma, the symbol for standard deviation.

1:14 And he had good data that showed that look, a normal distribution, that's the one that you see in the traditional bell curve right in the middle, that's how the world kind of sorts itself out, that if you were to give personal 1-to-1 to tutoring for students, then you could actually get a distribution that looks like that right. It says tutorial 1-to-1 with the asterisks, like, that right distribution, a two standard-deviation improvement. Just to put that in plain language, that could take your average student and turn them into an exceptional student.

1:37 It can take your below-average student and turn them into an above-average student. Now the reason why he framed it as a problem, was he said, well, this is all good, but how do you actually scale group instruction this way? How do you actually give it to everyone in an economic way? What I'm about to show you is I think the first moves towards doing that. Obviously, we've been trying to approximate it in some way at Khan Academy for over a decade now, but I think we're at the cusp of accelerating it dramatically.

2:02 I'm going to show you the early stages of what our AI, which we call Khanmigo, what it can now do and maybe a little bit of where it is actually going. So this right over here is a traditional exercise that you or many of your children might have seen on Khan Academy. But what's new is that little bot thing at the right. And we'll start by seeing one of the very important safeguards, which is the conversation is recorded and viewable by your teacher.

2:28 It’s moderated actually by a second AI. And also it does not tell you the answer. It is not a cheating tool. When the student says, "Tell me the answer," it says, "I'm your tutor. What do you think is the next step for solving the problem?" Now, if the student makes a mistake, and this will surprise people who think large language models are not good at mathematics, notice, not only does it notice the mistake, it asks the student to explain their reasoning, but it's actually doing what I would say, not just even an average tutor would do, but an excellent tutor would do.

2:50 It’s able to divine what is probably the misconception in that student’s mind, that they probably didn’t use the distributive property. Remember, we need to distribute the negative two to both the nine and the 2m inside of the parentheses. This to me is a very, very, very big deal. And it's not just in math. This is a computer programming exercise on Khan Academy, where the student needs to make the clouds part. And so we can see the student starts defining a variable, left X minus minus.

3:16 It only made the left cloud part. But then they can ask Khanmigo, what’s going on? Why is only the left cloud moving? And it understands the code. It knows all the context of what the student is doing, and it understands that those ellipses are there to draw clouds, which I think is kind of mind-blowing. And it says, "To make the right cloud move as well, try adding a line of code inside the draw function that increments the right X variable by one pixel in each frame." Now, this one is maybe even more amazing because we have a lot of math teachers.

4:10 We've all been trying to teach the world to code, but there aren't a lot of computing teachers out there. And what you just saw, even when I'm tutoring my kids, when they're learning to code, I can't help them this well, this fast, this is really going to be a super tutor. And it's not just exercises. It understands what you're watching. It understands the context of your video. It can answer the age-old question, “Why do I need to learn this?”