The old scaling paradigm

0:02 hey this is Lance from Lang Shain you've probably heard a lot about the new reasoning models from open AI such as 01 and 03 I want to talk about these by reviewing some of my favorite uh new videos and uh kind of blogs I've seen on this topic but first let's just start with the current scaling Paradigm that we've been in for a number of years which is next word prediction so Jason way has a great talk on this and he kind of frames nicely why has next word I works so well and he basically explains it as nexor prediction is a multitask learning problem when you ask an LM to predict the next word or the next token in a sentence it learns a lot of things at once it learns grammar it learns World Knowledge sentiment translation spatial reasoning math so this simple learning objective is extremely powerful and there's lots of nice papers and talks on this but that just sets the stage now we've scaled this over roughly seven orders of magnitude Jason also touches on this and this is well covered in the Capal at all paper from 2020 but with respect to model size data set size and train compute the overall capability of models trained just with

1:14 this kind of simple objective has gotten better now there's some interesting points here are related to the concept of emergence certain capabilities appear to be unlocked at certain scales like for example GPD 2 and three were poor at math four kind of unlock greater math capability but overall you can kind of think about the capability increasing in a Rel relatively predictable fashion with respect to size of model data set