Why evals matter now

0:00 The Metas and the Googles and all of the other large companies have to reinvent themselves right now in the age of AI. Every single PM is going to start building AI features. You cannot exist as a PM without understanding this. Meet Deno McKinnon, former PM on the Llama models at Meta, now a startup founder and a master of Evals. PMs right now from a career perspective are in a really tough situation. The average PM is an orchestrator, a motivator, and an analyst, but a lot of this is easy to do with AI.

0:38 What really is the difference between product management at Meta versus Google? Meta is like a much, much more aggressive culture. Uh, in many ways, Google is considered to be more of an engineering led company, whereas Meta is more of a product led company. If you're a PM who's never worked on an AI feature before, when would you be going through this Evals process? You work at Pinterest and you want to have, uh, better image generation that doesn't look like sloth and actually pleases users.

1:09 You need to think about from day zero. What does success look like? If it's the best way to measure it, we've got to learn it. Where should we start? Let me walk through. Before we get into today's show, please take a second to check that you're subscribed on YouTube and following on Apple and Spotify podcasts. If you want access to all of my favorite AI tools, I've gotten them to give you an entire year of their paid plans.

1:43 Check out bundle.akashg.com for an entire year of Bolt.new, Airtable, Speechify, Dict, Magic Patterns, Linear, Dovetail, Arise, and Mobbin. And now, into today's show. Daniel, welcome to the podcast. Yeah, thanks so much for having me and, uh, it'll be fun to talk about this stuff. So I want to start with your article. You had this provocative claim and this funny meme here. Do Evals replace the PRD? What is the role of Evals? Yeah, so I think it was like a little bit spicy and saying it replaces the PRD because without a product strategy or a particular customer, like your product is nothing.

2:27 But again, that's a paragraph or even a sentence, depending on what the product is. The majority of most PRDs that I've seen in my career have spent most of the document talking about specifics, how it will work, how it will behave in certain situations, how the user can expect to get value from that.