Education

Optimizing AI Agents: Prompts vs. Weights (GEPA, GRPO) | Stanford CS329Z Lecture 9 with Michael Ryan

by Stanford Online

Share:

📚 Main topics

  • When optimization is usefulOptimization requires a system, data that captures the task, and an evaluation metric. It is especially useful for adapting systems to a specific domain, improving reliability, and reducing cost or latency. 4:44
  • Context optimizationThe lecture covers few-shot example selection, bootstrapping successful reasoning traces, and reflective methods such as OPRO, MIPROv2, TextGrad, and GEPA. These approaches use evaluation results to search for prompts and examples that work well. 10:21
  • Weight optimizationSupervised fine-tuning trains on successful agent trajectories, while reinforcement learning uses task rewards to improve the student model. PPO and GRPO are compared, alongside on-policy distillation, which uses a teacher’s token-level feedback on student-generated outputs. 51:34
  • Agents as optimizersSystems such as the Darwin Gödel Machine and autoresearch let agents modify code or their own harnesses, run experiments, and use evaluation results to keep or discard changes. 15:30
  • Choosing an optimization strategyContext and weight optimization can complement each other. The right choice depends on model access, data and rollout costs, context length, and whether a capable teacher is available. 1:18:38

✨ Key takeaways

  • Evaluation makes intent actionableA good evaluation can express requirements that are difficult to specify completely in a prompt. Optimization then uses the evaluation to measure whether changes improve the system. 7:17
  • Context optimization can be sample-efficientPrompt and demonstration search can improve performance with fewer rollouts than reinforcement learning, making it useful when environment interactions are expensive. 1:19:10
  • GRPO avoids a value modelGRPO estimates advantage by comparing rewards across a group of outputs, reducing memory and training costs relative to PPO. 1:02:58
  • On-policy distillation gives dense feedbackThe student generates its own outputs and receives token-level guidance from a teacher, which can outperform SFT or RL in some settings. 1:11:18

🧠 Lessons learned

  • Successful traces can become demonstrationsBootstrapping keeps examples that score well and searches combinations of them, offering a practical way to build useful few-shot context. 19:09
  • Optimize for the actual metricEvaluation-driven methods can produce unusual prompts or behaviors, so metrics should reflect the system’s real requirements and be monitored for reward hacking. 35:52
  • Match the method to the model and dataContext optimization suits API-only models and frequent updates; weight optimization can help small models, distill capability into cheaper models, and avoid growing prompts. 1:19:42

🧠 Conclusion/next steps

  • Combine strategies when appropriateA system may benefit from optimizing prompts, then weights, and then prompts again. Compare methods using the same relevant evaluation and account for rollout and compute costs. 1:19:10
  • Treat evaluations as essential infrastructureWhether an agent is changing prompts, weights, or its own code, the quality of its metric and evaluation process is central to reliable improvement. 1:18:07

Transcript excerpt

0:05 Today's lecture is on optimization and I'm very excited to give this lecture because uh I if you recall a few weeks ago when I was presenting on DSPy I mentioned that optimization is the part of DSPy that I worked on most closely. So I'd say I have good background knowledge on some of this optimization stuff and I'm looking forward to teaching it today. Uh let's start with some announcements really quick. >> [snorts] >> So, first of all, guest lecture next Monday. Um, and so that will be on terminal bench. So, please be sure to

0:36 come. I think our we'll have two speakers and it should be a really, really engaging talk. Uh, so I actually have a fun story while I was preparing this lecture where I got to feel the AGI a little bit. I have some slides you'll notice that come from a talk that I gave for one of the papers that I'm presenting. Uh, but the format was completely different. And so I was like, oh, I was talking to D and I was like, do you think it's okay that the format is going to change mid-presentation? And she's like, why don't you just give this to Codex? Uh, so Codex went and reverse engineered I I do my presentations in Keynote. So Codex went

🔒 The full, searchable transcript is available with Pro.

🔒 Unlock Premium Features

This is a premium feature. Upgrade to unlock unlimited Q&A, transcripts, mindmaps, and translations.

Questions & Answers

Common questions about this video

What conditions are needed to optimize an AI system?

You need three things: a system to optimize, data such as rollouts or demonstrations, and a metric or evaluation benchmark to measure the effect of changes. 4:44

When is context optimization useful?

It is useful for last-mile adaptation when you have domain-specific data and evaluations that capture requirements that are difficult to specify manually. It can also help tune complex systems, improve reliability, and reduce cost or latency. 7:17

How does bootstrap few-shot optimization work?

The system generates candidate traces and keeps the ones that score well on a metric. It then tries different combinations of those successful examples in the prompt and evaluates them to find a useful few-shot set. 19:09

How does GEPA improve prompts?

GEPA evaluates prompts across tasks, keeps candidates on the Pareto frontier, and creates new candidates by reflecting on failures or merging useful parts of different prompts. It repeats this process within an optimization budget. 42:07

How does GRPO differ from PPO?

PPO uses a value model to estimate token-level advantages, while GRPO compares rewards across a group of sampled outputs and uses their normalized rewards as advantages. This removes the need for a value model and can make training more efficient. 1:02:58

What is on-policy distillation, and why might it help?

The student generates its own responses, and a teacher with accessible log probabilities provides dense, token-level learning signals for those responses. This can be more useful than learning only from teacher-generated traces, especially when the student’s outputs differ substantially from the teacher’s. 1:11:18

When should you choose context optimization over weight optimization?

Context optimization is a good fit when you use models that cannot be fine-tuned, need frequent updates, or want to save expensive environment interactions through sample-efficient optimization. Weight optimization may be preferable when prompts fail on smaller models, you have a capable teacher and want a cheaper model, or the context is becoming too long. 1:19:42

🔒 Unlock Premium Features

Access to Chat is a premium feature. Upgrade now to unlock unlimited studying tools.

🔒 Unlock Premium Features

Access to Mindmap is a premium feature. Upgrade now to unlock unlimited studying tools.

🔒 Unlock Premium Features

Access to Translation is a premium feature. Upgrade now to unlock unlimited studying tools.

Get unlimited summaries, Q&A, transcripts and more with Pro

Upgrade to Pro

Suggestions

🔒 Unlock Premium Features

Access to AI Suggestions is a premium feature. Upgrade now to unlock unlimited studying tools.