Measuring the Global State of AI

Editor-in-Chief, AI Index
Measuring the Global State of AI
Where is AI headed and how fast is it moving? Nestor Maslej breaks down the data behind the world’s most influential AI report to help make sense of what comes next.

00.07

All righty. Our next guest probably knows as much or more than anybody in the world about artificial intelligence. He's Nester Mazle, the former editor-inchief of the AI index report, published annually by Stanford's Institute for Human- Centered AI. It gives technologists, policy makers, executives, journalists, and the general public the best evidence-based assessment of where AI is and where it's going at any given point in time. His vantage point, unbiased and at the top of the AI food chain, is unparalleled. The AI index report is the gold standard annual assessment of AI progress. And as such, it helps decision makers assess AI based on facts and evidence, not on speculation or fear. Please welcome Nester Maslay, the former editor-inchief of the AI index report. Nester.

01.08

All right. I'm super excited to be here and really looking forward to chatting with you guys a little bit about what kind of things I see going on in the world of AI. And my talk today really much kind of grounded in what we're here to discuss and meditate on is thinking a little bit about the future of this technology. We're obviously in this post chat GPT moment, but where are we going to be going 10, 20, 30 years down the line? Now, a little bit about myself. uh I kind of cut my teeth in the AI world being the editor-inchief of I would say uh the world's best report on trends in AI at Stanford the AI index I've also taught a few courses at Stanford and I've contributed more recently to international reports like the international AI safety report which really aimed to kind of establish a consensus on this technology but it's really kind of with the index where I

02.01

cut my teeth this is a report that currently it's in its eth edition and it was created in 2017 by a diverse committee of AI thought leaders all of whom I've had the pleasure and privilege of collaborating with people like Jack Clark who's one of the co-founders of anthropic James Manika who currently is leading all AI related research activities at Google and people like Eric Bolson who I would say is one of the world's best AI economists and this report and the work that I've done we've been fortunate enough really to have been cited by newspapers across the globe, including newspapers like the New York Times, Bloomberg, Fortune. We've been cited by governments and government agencies across the globe. And I've also briefed business leaders across the globe. And I actually have recently founded my own company that does work educating the business leaders of tomorrow about this technology today. And I think this distinction, the fact

03.00

that AI is now something that's being discussed in boardrooms across the world, it's something that's being discussed in parliaments across the world. And quite frankly, it's something that's being discussed in kitchen tables across the world. Is really reflective of the fact that I think this is one of the most important technologies of the 21st century. And what I'm going to try to do today in 20 minutes is give you guys a sense and an ability to contribute to that discussion yourselves. So where is the future going? Now of course with AI you could talk about responsibility, you could talk about public opinion. I only have 20 minutes today. So I'm going to try to talk about three key points. The first being technical capabilities. What can the technology do? Well, we did an exercise at the AI index where we went to MidJourney, which is one of the world's best textto image generators, and we asked Midjourney to give us a hyperrealistic image of Harry Potter. And we asked each version of Midjourney

04.01

this question from one of the first versions that came out in 2022 to one of the more recent ones at the time that we did this work that came out in July of 2024. Now, you can see on this slide just how much better AI generations got in how little time. Now, I'm actually in the process of redecorating my apartment, and you know, I'm talking to my girlfriend about if we actually had to put one of these on our wall, which one would we go with? I mean, I think I'm voting for the one on the left. It looks like Pablo Picasso has drawn me a tortured Harry Potter, but again, we asked for hyperrealistic generations. And for the image on the right, you almost have to squint carefully to see that that is not in fact Daniel Ratcliffe. If you want kind of other views into dystopian futures of AI, you can go to websites like these which is called which faces.com. This website is going to give you an image of a human face that is artificially generated and an image that is an actual human. And then you can

05.01

vote on which one you think is real or not. And I'm gonna actually make you guys vote. Hands up if you think it is the image on the left that is the actual human. Hands up if you think it's the image on the right that's the actual human. So I think actually pretty 50/50. It is in fact the image on the left that is the real human. And for those of you that feel bad about yourselves, I mean to give you the extra kicker, this is old technology. We had this in 2020. So this is kind of really a sense of where this tech is. Or if you want kind of another view of this, you could also look at video generators. There was a very popular internet meme where people were asking video generators to give us an image of Will Smith, the famous American actor, eating pasta. And again, you know, I frankly I like postmodernism. I'm a lot more excited about the March 2023 generations than the one in January 2026. But this is again a scale and a view of how quickly and how rapidly this

06.01

technology is improving. So where does it actually leave us? Well, one of the exercises that we did at the index is think a lot about benchmarks. So ways in which we can measure technological progress. And what you see on this chart is probably one of the most famous images that we've produced at the index. And it's a chart that looks at how well AI systems do a variety of intellectual tasks, whether it is image classification, competition level math, or visual reasoning in comparison to humans. Now, you'll notice two things from this. On most of these tasks, AI systems already exceed humans. The human benchmark is at the 100 line. But that to me is not the main takeaway of this chart. The main takeaway is that all of these lines are becoming steeper and steeper. This benchmark on the left, that's a benchmark for image classification. And you'll notice that these AI systems, they went from roughly 90% relative to the human baseline all

07.01

the way to exceeding that baseline in four years. But that's for a basic benchmark that is looking at how well these systems can say this is a cat, this is a dog, this is an apple, this is an orange. One of these benchmarks on the far right, the benchmark in the teal, that's a benchmark that is looking at how well these systems can do competition level math. And we've gone from systems that scored roughly 10% relative to the human baseline all the way to systems that exceeded that baseline in pretty much two years. And the performance reference for that particular benchmark is the performance of a three-time international math Olympiad gold medalist. So, it's not like they're testing these systems against me. I still have nightmares about grade 12 AP calculus. They're testing these systems against some of the world's best mathematicians. So the key takeaway here when we talk about technology is that these systems are getting better, but how much better are they actually going to get? Right? We're trying to look into the future. Well, there's this well-known phenomenon in the world of AI

08.01

known as scaling laws. This is this idea that was discovered in 2020 that if you take an AI model and you pump more data into it, its performance will improve on a variety of tasks. And this story really starts with a transformer. You guys can think of the transformer as almost a new recipe for building AI systems. Google creates this recipe in 2017. And then eventually it's kind of realized that if you take this transformer-based model and you pump more data into it, you get systems that start to answer questions a lot better. And in the process, you're starting to get systems that require more and more computational resources because when you add more data, you need more compute. And you see in this chart how much more computationally expensive these systems have become. this chart and the one I'm about to show you now which looks at how expensive it has become to train these systems. Right? We're getting in the world of systems costing $200 million

09.00

perhaps costing even more. These two charts are the reason that Nvidia has become one of the world's richest companies because Nvidia functionally has a monopoly on the hardware technology that is required to power this scale up. So if you think about this logic of scale, maybe you're thinking to yourself, okay, we're going to pump more data into the systems and the systems are naturally going to get better over time. Well, there's other technologies like commercial airline cruising where we also saw this exponential rise in capabilities until this rise plateaued. And even if you look at some of the recent things that we've been hearing about GPT5, probably OpenAI's flagship model launch, there was a lot of negative reaction to it, right? people saying that it was a disaster. It's awful. It's OpenAI's worst release yet. We're also in a world where there are some concerns that quite frankly we're going to run out of data to train these systems. This chart is from some of my collaborators at Epoch

10.00

and they basically anticipate that we might get to a point in 2028 where we're not going to have any more data to continually scale these systems. And perhaps most importantly, and this touches on one of the big themes about this conference, innovation, AI is no longer as open as it once was. When GPT3, which was one of the chat GPT precursors, was published in 2020, OpenAI was very open about how it did the research, the architectures that it used, the hardware that it used. This was true open science in the world of AI. Fast forward to 2023. If you actually read the abstract of GPT4's launch, you'll notice the authors at OpenAI very explicitly say because of changes in the competitive landscape, we're no longer going to be as open about how we do science. And in a world potentially where these companies are sharing less and less information about how they do their work, one has to wonder if these technological

11.00

developments are going to continue. So, I told you guys that I would tell you about three things. The first one is the technology. The second one is geopolitics. Who's actually winning the AI race? And is this even a relevant question to ask? Well, it's really a twohorse story. It's the US and China. And there are certain metrics where the US is ahead. There are certain metrics where China is ahead. If you look at what countries are producing the most notable models, it's the United States that leads in this metric by a pretty substantial margin. If you look at what country is producing the research that gets the most cited, it's also the US that leads in this metric by a pretty wide margin. But China does really well when it comes to, for instance, patenting AI technologies. China also installs more industrial robotics than the rest of the world combined. And perhaps most relevant and this chart explains the story of

12.01

deepseek. Chinese models now have caught up to American models in terms of their technological capability. This is basically looking at how well AI models do on a very popular benchmark for assessing AI model capabilities, the chatbot arena. And you'll notice that in between January 2024 and February 2025, in January of 2024, there's a pretty big delta between the best US model and the best Chinese model. That gap closes pretty narrowly and the Chinese have caught up in terms of what they can do. So I think when you think about geopolitics, the key takeaway is that AI really is a two-horse race. Yet the question that I want you guys to meditate on is which country do you think is going to lead in 10 years? And I actually taught a course at Stanford in the continuing studies department on this very question. So I can't answer it in 20 minutes, but if you want my take, come talk to me after. But one thing

13.01

that I'll kind of give to maybe the business audience in the room is actually asking the question of would your businesses ever build with a Chinese model? When Deep Seek came out, there was a really interesting exercise some journalists did where they went to some of these leading models like Chad GPT like Gemini and Deepseek which is a Chinese model and they asked them what happened on June 4th 1989 which is when Teneman squared occurred and you'll notice chat GBT gives you an answer Gemini gives you an answer but Deepseek the Chinese model says sorry that's beyond my current scope. So there's important geopolitical implications when we think about what it might mean to eventually move to a world in which perhaps China has the best technology in class and what does it mean for America to to fall behind in that race. The last thing that I'll talk about is the impacts that AI has on the business world. Well, I think it's unquestionable

14.02

that in the last few years, AI has taken over the business world by storm. There is more investment privately in generative AI than ever before. More companies are also reporting using generative AI in their operations. We partner with McKenzie at the AI index and McKenzie has been doing this data since 2017 where they've been asking businesses, do you use AI in your operations? And this number kind of plateaued at around 50% from around 2018 to 2023. Now post chat GBT this number has shut up. We are in a world where now close to 90% of businesses are using this technology. And it's not just a story of business usage. If you actually look at developer revenues, generalpurpose AI technology is on pace to one of be the most lucrative technologies from a revenue perspective because again a lot of these companies aren't actually, you know, in

15.01

the green so to speak. And AI is being now adopted faster than any other technology. On this report that I worked on, the international AI safety report, we looked at where AI stood in comparison to technologies like the internet and the computer, and where it took roughly 15 years for around 50% of the population to use the computer, where it took roughly six to seven years for around 50% of the population to use the internet. We've gotten to that level with AI in pretty much two years. So in terms of how people are using the tech, it's also being adopted quicker than ever. And you have some of these new organizations that are doing work trying to assess how well these models work for magentic capabilities. And again, I think the story of this chart is really one of exponential growth. Right? The things that AI can do now technologically are nowhere near what it could do not even two decades ago. We're talking even two years ago. And I'm

16.01

going to kind of rifle through some of these slides. Again, I'm more than happy to share this afterwards, but the next few slides basically show that if you look at academic research, there is now a lot of cold hard evidence that AI is making a difference in the productivity of computer scientists. It's making a difference in the productivity of consultants and customer support agents. It's making a difference in the productivity impact of lawyers. And we now have these kind of macro studies where people are looking at a wide range of tasks, whether it's active listening to negotiating and seeing how much time does it take you to do with AI and how much time does it take you to do without AI. And for pretty much every single task that they've studied here, it's the AI users that are able to get it done in a lot less time. Now, all of these charts have led to these outsized predictions that AI is going to change the landscape of the economy. You have people like Eric Schmidt, former chairman of Alphabet and Google. He says

17.01

the AI revolution is under hype. Deis Hassabis who is the CEO of DeepMind which is the lab in Google that does a lot of high-tech research work says that AI is going to be 10 times bigger than the industrial revolution. So you have all this hype, yet on the other side of the foot, so to speak, we have these headlines like those that have come out in the New York Times in 2025 that companies are pouring billions into AI. Yet supposedly it hasn't paid off. There was a study that came out in August of 2025 that said that 95% of generative AI pilots at companies are failing. So what's really going on here? Is AI this transformative technology or is it something that is going to be the next Silicon Valley fad? And I want to close in the next three to four minutes with what I think is the most important takeaway from this presentation. This explains why I've kind of founded this

18.01

business and it's this logic that it takes time for technologies to make an impact in the business world. This is a provocative chart. It's looking at global GDP over the long run from the year Jesus Christ was born to the present. And you'll notice this is pretty much a flat horizontal line until something happens in 1712. And that's the introduction of the steam engine. When steam is introduced and the industrial revolution start, this line that is once flat horizontally goes through the roof vertically. And when people talk about AI, they also conceptualize AI as this generalpurpose technology. Not a generalpurpose transformer, which is what the GPT stands for in chat GPT, but a technology like rails, steam, or electricity that is pervasive, improving, and complimentary. Now, funnily enough, when these previous GPTs came out, it actually took a lot of

19.02

time for these technologies to have broad-based economic impacts. This is a chart from research work that economists like David have done where they look at how long it takes for new technologies to start moving the needle productivity-wise. And basically this economist estimates that with a technology like it or portable power it took roughly 25 years for these technologies to break even productivity wise. Now the reason this is the case is you can see this story very well when you think about how we used to operate factories in the world of steam and how things eventually changed. When you have steam factories, you had factories that had central large steam engines and they would power transmission through the factory with these systems of belts, pulleys and gears. Now when electric technology was developed, what a lot of factory managers did is they basically took out these central large single steam engines

20.01

and they replaced them onetoone with electric engines. Now, when they replaced them onetoone with electric engines, they actually didn't find that the productivity went up three to four times. It was only until a lot of these factory managers thought to themselves, we could actually completely reorganize how a lot of these factories are designed that you started to see productivity really go through the roof. More specifically, what a lot of these factory managers did is instead of having factories organized around the transmission of power with electricity, they could have several small decentralized engines, you could now organize factory labor around optimal workflow instead of optimal power transition. And in the process of this transformation, that's when you start to see productivity go up three to four times. So, I'll kind of say in conclusion that I really think that AI is one of the most fundamental

21.01

technologies of the 21st century. I've only talked to you guys today about where the tech is going, where the geopolitics is going, how it might be changing the economy, but what I would encourage all of you guys to do is to make it your mission to involve yourself in the dialogues that surround this technology. I think any technology is neither good nor bad but the future that it is ultimately going to have depends on the voices at the table and it is important that all of you guys think about what your voice is going to be in the world of AI. That's the work that we really did at the index trying to empower people to find their voice with this technology. And with that I'll close and say a big thank you for having me speak today. Thank you very much.