Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)
📚 Main topics
- How frontier models are builtModel development combines careful research and monitoring with unpredictable outcomes. A PM needs to understand both what the team set out to build and the surprising behaviors that emerge. 3:04
- Google’s full-stack AI strategyGoogle aims to compete at the frontier while building a range of models and experiences, from efficient systems to products serving billions of users and specialized needs. 7:10
- Safety and cybersecuritySafety is built into training and deployment, with monitoring helping identify misalignment. AI can also help defenders find and patch vulnerabilities in real-world systems. 10:21
- Product experience and model behaviorDifferent users need different interfaces, and the model’s communication style, transparency, and personality are part of its UX. PMs work with researchers to define desired behavior and turn it into rubrics and training data. 17:04
- PMs’ growing valueAs AI makes building easier, PMs can provide taste, identify valuable problems, distinguish signal from noise, and bring clarity to teams moving quickly. 19:41
- Evals, vibes, and end-to-end qualityPMs should develop intuition by testing models on tasks they care about, while also creating structured evaluations. Evaluation should cover the judge, serving setup, safeguards, and the complete user experience. 23:17
- Building for changing capabilitiesDoshi’s 60-20-20 framework suggests building something that works now, something that works well enough as models improve, and something ambitious that could become possible soon. 33:38
- Helping teams adaptTeams can share useful workflows and technical skills, run workshops, and make time for experimentation. A simple habit is to pause before doing a task and ask whether AI can help. 37:46
- AI, workload, and burnoutAI may reduce repetitive work and make room for more creative tasks, but teams still need to manage chaos, workload, and individual motivations. 45:07
- How AI is changing PM workPrototyping and data analysis are more accessible, and the PM role increasingly involves adding substance beyond written requirements—through prototypes, evals, or data. 49:47
✨ Key takeaways
- Build intuition through hands-on useStrong research PMs use models regularly, form opinions about their behavior, stress-test them, examine data, and adapt as the field changes. 21:44
- Start evals smallBegin with a handful of prompts, then expand to around 20 that reveal meaningful differences between models. Even judging examples yourself can clarify what “good” means. 23:17
- Choose tests you care aboutUse personally meaningful tasks—such as writing, visual creation, or analysis—to notice where a model is improving and where it still falls short. 24:52
- Revisit outdated evaluationsThe Fitzpatrick scale did not represent the full range of global skin tones, so Google incorporated the Monk Skin Tone scale into evaluations and used it to improve products. 26:54
- Keep products adaptableModels can improve beyond the assumptions built into existing systems. Retest them and avoid forcing new capabilities into outdated structures. 35:43
- Separate insight from generated materialAI can efficiently gather and summarize data for documents, while people should contribute the insight and judgment. 48:13
🧠 Lessons learned
- Expect surprises from modelsSet clear goals and monitor progress, but leave room for capabilities and behaviors that emerge unexpectedly. 3:04
- Balance present and futureDon’t build only for current capabilities or bet entirely on a forecast; combine working features with near-term and more ambitious opportunities. 34:09
- Make space to learnWorkshops, shared skills, and dedicated time to experiment can help teams build confidence with AI tools. 39:20
- Look for meaningful workBurnout can come from chaos and tedious work, not simply long hours. Removing friction and aligning work with what motivates each person may help. 45:38
- Try AI once, then build on what worksOne useful example can increase confidence and encourage people to find more tasks where models can help. 41:58
🏁 Conclusion/next steps
- Use and evaluate Google’s modelsDoshi invites users to try Gemini and Google’s other models, including audio and multimodal products, and share feedback. 57:04
- Keep experimenting and adaptingBuild products that work today while staying ready for model improvements, and give teams the time to learn, test, and refine their approach. 34:09
Transcript excerpt
0:00 The model isn't predictable. It's kind of like Snape's potions class. You can put all the ingredients in, but you also have to expect that there will be a little bit of magic, and that magic is going to be surprising. >> As you're hiring PMs these days, is there anything you look for more and more? >> We are looking for PMs who understand the vibes of our models or other models. I want PMs who [music] have an opinion, who are playing with these models every day. You ideally want to stress test the model on things that you are excited about. A lot of people feel that Google
0:30 lags behind the top two labs right now. >> I actually [music] think we do want to win the frontier conversation. I don't think we should say that Google is not participating in the frontier. >> You have this framework around how far ahead to build. You call it the 602020 rule. >> Traditional startup wisdom. You're just like build an MVP that works. But I think you want to actually build features that aren't necessarily always working today. It feels like every AI assistant launching these days is taking all this juicy Google data like Gmail, Google Calendar, my Docs. I'm curious
🔒 The full, searchable transcript is available with Pro.
Questions & Answers
Common questions about this video
¿Por qué Tulsee Doshi compara el desarrollo de modelos con la clase de pociones de Snape?
Porque, aunque los equipos establecen objetivos y controlan el entrenamiento, los modelos no son totalmente predecibles: pueden mostrar capacidades inesperadas. El trabajo del PM consiste también en descubrir dónde destaca el modelo y convertir esas cualidades en experiencias valiosas para los usuarios. 3:04
¿Qué significa que Google tenga una estrategia de IA de «stack completo»?
Significa que Google trabaja desde el hardware hasta los modelos y los productos finales, que llegan a una enorme cantidad de usuarios. La meta es crear experiencias de frontera y modelos con distintos niveles de coste y calidad para una variedad de necesidades y usuarios. 7:10
¿Por qué, según Tulsee, los PM están volviendo a ser especialmente valiosos?
Como ahora es más fácil construir, también es fácil producir cosas que no resuelven problemas reales. Los buenos PM aportan criterio para separar la señal del ruido, identificar qué merece la pena construir y ayudar a los equipos a ponerse de acuerdo y convertir el caos en claridad. 20:12
¿Qué habilidades busca Tulsee en los PM que trabajan con modelos de IA?
Busca PM que experimenten con los modelos a diario y desarrollen criterio sobre su comportamiento; que puedan analizar datos, crear evaluaciones y ponerse manos a la obra; y que sean ágiles para revisar sus planes a medida que el campo cambia. 21:44
¿Cómo recomienda empezar a crear una evaluación para un modelo?
Empezar en pequeño: reunir unos pocos prompts —por ejemplo, unos 20— que pongan a prueba las capacidades relevantes, y evaluar las respuestas uno mismo. Aclarar primero qué significa una respuesta buena ayuda a reconocer señales y luego permite ampliar la evaluación. 23:17
¿En qué consiste la regla 60-20-20 para desarrollar productos con IA?
La idea es que alrededor del 60 % del producto funcione ahora, un 20 % sea imperfecto pero parezca alcanzable pronto, y el 20 % restante represente una experiencia ambiciosa para la que ya se ven indicios. Así el producto es útil hoy y puede crecer junto con los modelos. 33:38
¿Qué aspectos hay que considerar para evaluar la experiencia completa de un modelo?
No basta con evaluar el modelo: también importa la calidad del sistema que lo juzga, las mitigaciones y filtros de producción, y la forma en que se sirve. En audio, por ejemplo, hay que valorar no solo el contenido, sino también la naturalidad, el tono y las inflexiones. 28:26
¿Cómo propone Tulsee repartir el trabajo entre la IA y las personas al crear documentos?
Sugiere usar la IA para recopilar y resumir datos, como métricas, comentarios y tendencias, y dejar que las personas aporten el criterio y las ideas. En sus palabras, la persona debería aportar la perspectiva; el modelo, los datos. 48:13
Related summaries
86 minutes of paid-growth alpha | Hector Serrano
An 86-minute guide to finding PMF | Rob Snyder (Author, The Power of Pull)
A mind-expanding conversation on how models actually work | Emmanuel Ameisen (Anthropic)
Why every company now needs to think and operate like a lab team | Josh Woodward (VP Google Labs)
We replaced our sales team with 20 AI agents—here’s what happened next | Jason Lemkin (SaaStr)
