What You Will Learn

• Implement ReAct loops to ground LLM reasoning with tool actions and observations
• Apply RLEF to run, test, and iteratively repair code using execution feedback and public/private tests
• Use Constitutional AI to generate principled self-critiques and train aligned, harmless responses
• Combine tool, execution, and AI-critique feedback to build interpretable self-improving agents
• Design practical feedback pipelines: small action spaces, logging, reflection, and noisy-observation handling

Study Guide

# Self-Improving AI Agents: Learning from Feedback with Tools, Code, and Principles ## Introduction: Why This Course Matters Let me start with a confession. When I first started working with large language models, I was blown away by what they could do with just a prompt. Ask them to write a poem, solve a math problem, draft an email,done. But then came the reality check. Ask them something that requires current information, and they'd confidently make things up. Ask them to actually *do* something in the world, and they'd freeze. Ask them to write code that actually runs, and they'd produce elegant-looking nonsense. The fundamental problem is that a language model, by itself, is like a brilliant scholar who's been locked in a room with no windows. They know a lot, but they have no way to check anything, no way to test their ideas, and no feedback loop to tell them when they're wrong. This course is about breaking down those walls. We're going to explore three major paradigms that transform static language models into interactive, self-improving agents. These aren't just theoretical concepts,they're practical frameworks that are reshaping how AI systems are built and deployed in the real world. The first paradigm, ReAct, shows us how to combine reasoning with action, letting models think and then check their thinking against external tools. The second, RLEF, demonstrates how execution feedback,actually running code and seeing what breaks,can dramatically improve code generation. The third, Constitutional AI, tackles the thorny problem of alignment, showing how AI systems can learn to be both helpful and harmless through principled self-critique rather than massive human labeling efforts. By the end of this guide, you'll understand not just what these approaches are, but why they work, where they fail, and how you can apply these principles to build better AI systems. Let's dive in. ## Part 1: The Problem with Static Language Models ### What LLMs Are Actually Good At Large language models are remarkable at processing and generating text. They can summarize documents, answer questions about topics in their training data, translate between languages, explain complex concepts, and even engage in surprisingly sophisticated reasoning. When you give them a math word problem and ask them to think step by step, they often produce correct solutions with clear explanations. This is genuinely impressive. The chain-of-thought prompting technique, where you ask the model to reason through a problem before answering, unlocked capabilities that nobody expected from pure text prediction. Models could solve multi-step reasoning problems by breaking them down into intermediate steps, much like a human working through a problem on a whiteboard. ### Where They Fall Apart But here's where things get messy. These same models, despite their brilliance, have critical limitations that prevent them from being truly useful in real-world applications. First, they hallucinate. When a model doesn't know something, it doesn't say "I don't know",it makes something up with complete confidence. Ask it about a current event that happened after its training cutoff, and it will invent plausible-sounding but completely fabricated details. This isn't a bug; it's a fundamental consequence of how these models work. They're predicting the most likely next token based on patterns in their training data, not accessing a database of facts. Second, they have no notion of grounding in the real world. A model might know that Paris is the capital of France, but it can't check the current weather in Paris, look up the latest news, or verify whether a claim is true. It's operating entirely on its internal knowledge, which is frozen at the time of training. Third, and this is crucial for our purposes, they can't learn from experience. If a model generates wrong code, it doesn't know the code was wrong unless someone tells it. If it gives a harmful response, it doesn't get feedback that it should have been more careful. The model is static,it does the same thing every time you give it the same input, regardless of whether the output was good or bad. ### The Feedback Gap So what's missing? Feedback. Real learning,whether in humans or AI systems,requires a feedback loop. You try something, you see what happens, and you adjust your approach based on the outcome. This is how a child learns to ride a bike, how a chef learns to cook, how a programmer learns to debug. Traditional language models have no such loop. They generate text, and that's the end of the story. They never see the consequences of their outputs. They never learn whether their answers were correct, their code compiled, or their advice was helpful. The solution, as we'll explore throughout this course, is to create feedback channels. Three major approaches have emerged, each with its own source of feedback: - **Environmental feedback**: Tools and external systems that provide observations (ReAct) - **Execution feedback**: Running code and seeing what happens (RLEF) - **Principled AI feedback**: Self-critique guided by explicit values (Constitutional AI) Each of these approaches addresses a different aspect of the feedback problem, and together they show us how to build agents that genuinely improve over time. ## Part 2: ReAct,Combining Reasoning and Action ### The Two Historical Approaches Before ReAct came along, there were two separate lines of research trying to make language models more capable. The first was chain-of-thought prompting. Give the model a problem, ask it to "think step by step," and it produces explicit reasoning traces before arriving at an answer. This approach improved performance on many reasoning tasks, but it had a critical weakness: all the reasoning happened internally. The model was thinking, but it wasn't checking anything. Its reasoning could be completely wrong, and there was no way to catch the error because there was no external validation. The second approach was action-focused. Models like WebGPT were trained to interact with a browser, clicking links and reading pages. These models were grounded in real data,they could access current information,but they didn't reason explicitly about what they were doing. Their decision-making process was opaque, and they often made poor choices about what to search for or click on. The insight behind ReAct is beautifully simple: why not do both? Why not have the model alternate between explicit reasoning and concrete actions, using the results of its actions to inform its reasoning, and its reasoning to guide its actions? ### The Thought-Action-Observation Loop The ReAct paradigm is built around a simple loop: **Thought → Action → Observation → Thought → Action → Observation → ...** Here's how it works in practice. The model is given a task and a set of available actions. At each step, it produces a thought,a natural language reasoning step that explains what it's thinking and what it plans to do next. Then it produces an action,a specific tool call from its predefined action set. The tool executes and returns an observation. The model reads the observation, updates its thinking, and decides on the next action. This loop continues until the model decides it has enough information to produce a final answer, at which point it emits a special "Finish" action with its response. The key insight is that this entire process happens in language space. Thoughts are just text tokens. Actions are text commands that get parsed and executed by the system. Observations are text returned by the tools. This means we can implement ReAct entirely through prompting,no special training required. ### Action Spaces: The Guardrails One critical design decision in ReAct-style systems is the action space. You can't just let the model do anything. Instead, you define a finite set of valid actions that the model can choose from. For a question-answering system, this might be: - `Search[query]` , search Wikipedia for information - `Lookup[page_title]` , open a specific page - `Finish[answer]` , provide the final answer For a web navigation agent, it might be: - `Search[query]` , search for products - `Click[item]` , click on a specific item - `GoBack[]` , return to previous page - `AddToCart[item]` , add an item to the cart - `Checkout[]` , proceed to checkout By constraining the action space, we accomplish several things. First, we prevent the model from taking invalid or dangerous actions. Second, we make the decision problem tractable,instead of generating arbitrary text, the model is essentially doing classification: given the current context, which action from this finite set should I take? Third, we make the system auditable,every action is a discrete, logged event. This is especially important in domains like robotics, where an invalid action could cause physical harm. If a robot controller can only choose from a predefined set of safe movements, the risk of catastrophic failure is dramatically reduced. ### A Concrete Example: The Apple Remote Question Let me walk through a detailed example to make this concrete. This is a classic multi-hop question answering task from the HotpotQA benchmark: "Aside from the Apple Remote, what other device can control the program the Apple Remote was originally designed to interact with?" This is a multi-hop question because you can't answer it with a single search. You need to first find out what program the Apple Remote was designed to control, and then find out what other devices can control that program. **Standard prompting** would give the model this question and ask for an answer directly. The model might guess, might hallucinate, or might just say something plausible-sounding but wrong. **Chain-of-thought prompting** would ask the model to reason step by step. The model might say something like, "The Apple Remote was introduced in 2005. It was designed to control multimedia software on Mac computers. The most likely program is Front Row. Other devices that can control Front Row include..." But here's the problem,the model is reasoning entirely from its internal knowledge. If it doesn't actually know what program the Apple Remote was designed for, or what other devices can control it, all the reasoning in the world won't help. It'll just produce confident-sounding nonsense. **Action-only** would have the model issue searches without explicit reasoning. It might search for "Apple Remote" and "Front Row" but without a clear strategy for what to search next or how to interpret the results. It might get lost or miss the key connection. **ReAct** interleaves reasoning and action: 1. **Thought**: "I need to find out what program the Apple Remote was originally designed to control. Let me search for information about the Apple Remote." 2. **Action**: `Search[Apple Remote]` 3. **Observation**: The search results tell us the Apple Remote was introduced in 2005 and was designed to control the Front Row media center software on Mac computers. 4. **Thought**: "The Apple Remote was designed for Front Row. Now I need to find out what other devices can control Front Row. Let me search for that." 5. **Action**: `Search[Front Row media center]` 6. **Observation**: The results indicate that Front Row could be controlled by various devices, possibly including the iPhone or iPod Touch through remote applications. 7. **Thought**: "I have enough information. The answer is the iPhone (or iPod Touch)." 8. **Action**: `Finish[The iPhone can also control the program the Apple Remote was originally designed to interact with.]` The key difference here is that the reasoning isn't happening in a vacuum. Each thought is grounded in actual observations from the environment. When the model searches and gets results, those results shape what it thinks and does next. If the search results are surprising or contradictory, the model can adjust its approach. ### Why ReAct Reduces Hallucination The hallucination problem in language models comes from the fact that they're generating text based on patterns, not facts. When a model doesn't know something, it makes something up because that's what it's trained to do,produce plausible text. ReAct addresses this by giving the model a way to check its knowledge. Instead of relying on internal knowledge, the model can search for information, read the results, and base its answers on what it actually finds. This doesn't eliminate hallucination entirely,the model could still misinterpret search results or make errors in reasoning,but it dramatically reduces the most common failure mode of confidently asserting false information. ### Benchmarks and Results ReAct has been evaluated on several challenging benchmarks: **HotpotQA** is a multi-hop question answering task where questions require information from multiple Wikipedia articles. The model needs to find and combine information from several sources to answer correctly. **FEVER** is a fact verification task where the model must determine whether a claim is supported, refuted, or not enough information is available, based on Wikipedia evidence. **WebShop** is a more complex decision-making environment where the agent must navigate an online store, search for products, compare options, and make purchases based on natural language instructions. The results are instructive. On knowledge-intensive tasks like HotpotQA and FEVER, ReAct generally outperforms pure action-only approaches, showing that explicit reasoning improves tool use. The model knows what to search for and how to interpret results because it's thinking about the task, not just mechanically issuing queries. Compared to chain-of-thought, the picture is more nuanced. On FEVER, ReAct outperforms CoT because factual grounding matters more than reasoning ability. On HotpotQA, CoT sometimes matches or slightly exceeds ReAct. But here's the key finding: hybrid strategies that combine both approaches achieve the best performance. You can use ReAct to gather information, then use CoT with self-consistency to reason over what you've found, or use ReAct as a fallback when CoT fails. On WebShop, ReAct-based agents achieve higher scores and higher success rates than pure imitation learning or imitation learning plus reinforcement learning. But there's still a significant gap to human performance,ReAct achieves a score of about 66.6, while human experts achieve around 82.1. This gap represents an opportunity for further improvement. ### When ReAct Struggles ReAct isn't perfect. Let's talk about its failure modes. First, there's the inference cost problem. Every thought-action-observation cycle adds tokens to the context and requires additional API calls. A simple question that could be answered in one step might take five or six steps with ReAct. This increases latency and cost, which can be prohibitive for production applications. Second, there's the action space limitation. When you have a small action space (like search, lookup, finish), ReAct works beautifully. But when you have hundreds or thousands of possible actions, the model needs many demonstrations to learn when to use each one. These demonstrations might not fit in the context window. Third, there's the overthinking problem. Some modern models, when given the ReAct framework, produce excessively long reasoning traces even for simple tasks. They think, and think, and think, generating elaborate reasoning chains for questions that could be answered in one step. This is both wasteful and error-prone,more reasoning steps mean more opportunities for errors to creep in. Fourth, there's the noisy environment problem. What happens when search results are contradictory or misleading? The model might follow a wrong trail and end up with an incorrect answer. ReAct doesn't inherently handle this well. You need additional mechanisms like backtracking, repeated retrieval with voting, or reflection steps where the model critiques its own trajectory. ### Practical Tips for Implementing ReAct If you're building a ReAct-style agent, here are some lessons from real-world implementations: **Start with a small action space.** Don't give your agent fifty tools when it only needs five. A smaller action space means fewer demonstrations needed and fewer opportunities for error. **Log everything.** The thought-action-observation traces are gold. They let you see exactly where the agent went wrong and fix the problem, whether it's a prompt issue, a tool issue, or a reasoning issue. **Handle noisy observations.** Don't just feed raw tool output back to the model. Clean it up, extract relevant information, and present it in a structured way. **Add a reflection mechanism.** After the agent finishes, have it review its own trajectory and consider whether it missed anything. This catches many errors that would otherwise slip through. **Consider hybrid approaches.** Don't force every task through the ReAct loop. For simple questions, direct prompting might work fine. Use ReAct when you need grounding, and fall back to simpler approaches when you don't. ## Part 3: RLEF,Grounding Code LLMs in Execution Feedback ### The Challenge of Code Generation Let's shift gears to a different but related problem: code generation. Language models have become remarkably good at writing code. Give them a natural language description of a programming task, and they'll produce a function that looks plausible. The problem is that "plausible" isn't the same as "correct." Anyone who's worked with AI code assistants knows the experience. The model generates code that looks right,proper syntax, reasonable variable names, sensible structure,but when you run it, it fails. Maybe there's a logic error. Maybe it times out on certain inputs. Maybe it produces wrong outputs for edge cases. The fundamental issue is that the model is generating code based on patterns in its training data, not based on whether the code actually works. It has no way to test its own output. It doesn't know if the code compiles, if it passes tests, or if it's efficient enough. ### The Core Idea: Execution as Feedback The RLEF (Reinforcement Learning from Execution Feedback) framework addresses this by making execution feedback a central part of both the inference process and the training process. The insight is simple: when you're writing code, the ultimate judge is whether the code works. Not whether it looks good, not whether it matches some style guide, but whether it produces correct results when executed. So why not use execution results as feedback for the model? The framework works like this: 1. The model receives a natural language problem description,for example, a competitive programming problem. 2. The model generates a code solution. 3. The solution is executed against a set of public tests. 4. If the code fails, the failure messages, error traces, and timeout information are fed back to the model. 5. The model generates a revised solution, incorporating this feedback. 6. This loop continues until the solution passes the public tests or a maximum number of iterations is reached. But this is just the inference-time loop. The real power of RLEF comes from the training-time loop. ### The Training-Time Loop After the model has iteratively refined its solution and passed the public tests, the solution is evaluated against a private test set,tests that were held out and not seen during the generation process. The pass/fail results on these private tests become the reward signal for reinforcement learning. The model is updated using PPO (Proximal Policy Optimization), a standard RL algorithm. Over many problems, the model learns patterns: - Which types of solutions tend to be correct? - What kinds of errors are common, and how to fix them? - How to interpret failure messages and use them to guide repairs? This is the self-improvement loop. The model doesn't just learn to generate better code from scratch; it learns to iterate, to diagnose its own errors, and to fix them. ### The Two-Tier Test Strategy One of the most important design decisions in RLEF is the separation of tests into public and private sets. Public tests are the tests the model can see and use during its iterative refinement loop. They provide immediate, granular feedback. If the model's code fails a public test, it sees exactly which test failed and why. This guides the repair process. Private tests are hidden. The model never sees them during generation. They're used only to compute the reward signal for RL training. Why this separation? If you let the model see all the tests, it could overfit,memorizing specific test cases and generating code that passes those exact tests without generalizing to the underlying problem. The private tests ensure that the model actually learns to solve the problem, not just to pass the specific tests it was shown. This two-tier approach also mirrors real-world software development. When you're writing code, you have your own tests to guide you, but the real evaluation happens when someone else runs their tests against your code. ### The Hybrid Policy-Value Architecture Another interesting design element is the hybrid token-level policy and turn-level value architecture. The policy model,the language model that generates code,operates at the token level. It generates code one token at a time, giving it fine-grained control over the output. This is natural for language models, which are fundamentally token prediction machines. But the value function operates at the turn level. Instead of assigning a value to each token, the system computes a single value for the entire generated solution. This value is then used to compute an advantage signal that's applied to all tokens in the turn. This is similar to sequence-level reward methods in other RL-for-language work. The idea is that you can't meaningfully assign credit to individual tokens,the correctness of a solution depends on the whole, not on any particular token. So you evaluate the whole solution and use that evaluation to update the policy. ### A Concrete Example: The Palindrome Problem Let me walk through a concrete example to make this framework tangible. Consider a problem: write a function that finds all palindromic substrings in a given string. **Turn 1**: The model generates an initial solution. It might use a naive approach,triple nested loops that check every possible substring. The code looks correct, but when executed against the public tests, it fails. Specifically, it times out on a test with a long input string. The execution feedback includes the timeout error and possibly the specific test case that triggered it. **Turn 2**: The model receives this feedback. It can see that the problem is performance-related, not logic-related. It knows its algorithm is too slow. So it generates a revised solution using a more efficient approach,perhaps the expand-around-center algorithm, which runs in O(n²) time instead of O(n³). This solution passes the public tests. **Private evaluation**: The solution is now evaluated against the private test set. If it passes, the model receives a positive reward. If it fails, a negative reward. This reward is used to update the policy. Over thousands of such problems, the model learns patterns. It learns that naive algorithms often time out. It learns that when it sees a timeout error, it should look for algorithmic inefficiencies. It learns which types of solutions tend to be correct and which tend to be wrong. ### The Evidence: What Actually Improves The key finding from RLEF research is that execution feedback genuinely drives improvement. But it's not just about having feedback,it's about how you use it. One of the most interesting results is that base models given access to faulty solutions and raw execution logs don't automatically benefit. If you just take a pre-trained model and show it error messages, it tends to repeat the same mistakes. The combination of turn-by-turn feedback from public tests AND policy updates based on private test outcomes is what drives improvement. Across turns, the improvements are measurable: - The number of wrong outputs decreases as the model iterates. - The proportion of targeted code changes,changes that actually fix bugs,increases. - The model becomes better at interpreting error signals, including timeouts, as guides for repair rather than just opaque failure indicators. Models trained with RLEF show higher solve rates across validation and test sets compared to baseline models, even when evaluated on a log-scale sampling budget. This means that the model doesn't just get lucky more often,it genuinely produces better solutions. ### The Generalization Question One of the most exciting aspects of RLEF is that the improvements generalize. Models trained on competitive programming problems from one benchmark show improvement on other code-generation benchmarks. This suggests that the model is learning something fundamental about how to write correct code, not just how to solve specific problems. This makes sense when you think about what the model is learning. It's not memorizing solutions to specific problems. It's learning patterns like: - "When I see a timeout error, I should look for algorithmic inefficiencies." - "When my code produces wrong outputs, I should check edge cases." - "When a test fails, I should read the error message carefully and identify the specific issue." These are transferable skills that apply to any coding task. ### Limitations and Open Questions RLEF isn't a silver bullet. Let's talk about where it falls short. First, binary pass/fail rewards are coarse. For simple problems, they might be sufficient. But for complex tasks, richer feedback would be more useful. Imagine if the model could see partial credit for partially passing tests, or structured hints about what went wrong, or information from static analyzers about potential issues. This is an active area of research. Second, the approach can encourage trial-and-error repair rather than first-try correctness. If the model knows it can iterate and fix its mistakes, it might not put as much effort into getting things right the first time. Balancing these objectives requires careful reward shaping. Third, scalability to large codebases is a challenge. The RLEF framework works well for self-contained problems that can fit in the context window,competitive programming problems, small functions, etc. But real-world projects have thousands of files, complex dependencies, and context that exceeds what any language model can handle. To scale, you need additional mechanisms: - **Search-based retrieval**: Find relevant files and functions using textual or structural search, and feed only those into the context. - **Summarization**: Generate summaries of modules or classes, and use those compressed representations in context. - **Graph-based representations**: Build a graph over functions and files (call graphs, import graphs), and retrieve subgraphs relevant to the current task. Fourth, there's the unresolved question of process rewards versus outcome rewards. Should the model get feedback on intermediate steps, or only on final correctness? For complex reasoning or long code, process rewards might be more informative. But they're also harder to define and compute. ### Practical Tips for Using Execution Feedback If you're building coding agents or AI programming assistants, here are some practical recommendations: **Start with a strong test suite.** The quality of your execution feedback depends entirely on the quality of your tests. Good tests catch real bugs and provide useful error messages. **Structure the feedback.** Don't just dump raw error logs into the prompt. Extract the relevant information: which test failed, what was the expected output, what was the actual output, what was the error type. **Limit the iteration count.** You don't want the model to loop forever. Set a maximum number of turns, and after that, accept whatever solution you have or escalate to a human. **Monitor repair quality.** Track whether the model's revisions actually fix bugs or just shuffle code around. If you see the model making the same mistake repeatedly, it might need better feedback or more training. **Consider richer feedback signals.** Binary pass/fail is a starting point. Error traces, partial credit, performance metrics, and static analysis warnings can all provide more guidance. ## Part 4: Constitutional AI,Learning from Principled AI Feedback ### The RLHF Bottleneck Let's talk about alignment,the process of making AI systems behave in ways that are helpful, safe, and aligned with human values. The standard approach is RLHF (Reinforcement Learning from Human Feedback). Here's how it works: 1. A base language model generates responses to various prompts. 2. Human annotators compare pairs of responses and rank which one is better,more helpful, more accurate, more appropriate. 3. These human preferences are used to train a reward model, which learns to predict which responses humans would prefer. 4. The base model is fine-tuned using reinforcement learning to maximize the reward model's score. RLHF has been remarkably effective. Models trained with RLHF are significantly more helpful, more aligned, and more useful than their base counterparts. But it has a critical bottleneck: human labor. Training a reward model requires "tens of thousands" of human preference labels. Each label requires a human to read two responses and decide which is better. This is time-consuming, expensive, and difficult to scale. Every new domain, every new safety requirement, every new behavioral specification might require new data collection. ### The Constitutional AI Alternative Constitutional AI offers a different approach. Instead of relying primarily on human preference labels, it uses AI-generated feedback guided by a human-written "Constitution" of principles. The Constitution is a set of natural language principles that define desired behavior. These principles might include: - Avoid providing harmful or unethical instructions. - Avoid gender, racial, or other demographic biases. - Ensure responses are appropriate for young audiences. - Be respectful, thoughtful, and cordial. These aren't vague aspirations,they're operational principles that can be used to critique and revise model outputs. ### Stage 1: Supervised Self-Critique and Revision The first stage of Constitutional AI is supervised learning through self-critique and revision. Here's how it works: 1. The system generates "red-teaming prompts",prompts designed to elicit problematic behavior. These might be requests for harmful advice, biased responses, or inappropriate content. 2. The model generates an initial response to these prompts. 3. The model is then prompted to critique its own response using the Constitution as a guide. The critique prompt might say something like: "Does this response contain anything harmful or unethical? If so, explain why." 4. The model generates a critique, identifying problematic elements in its response. 5. The model is then prompted to revise its response: "Rewrite the response to remove anything harmful or unethical while preserving as much helpful content as possible." 6. The revised responses are used as training targets for supervised fine-tuning. This process is repeated many times, and the model learns to generate responses that align with the Constitution. The key insight is that the model itself is doing most of the work. It's generating the initial responses, critiquing them, and revising them. The human role is limited to writing the Constitution and selecting the red-teaming prompts. ### The Trade-Off: Helpfulness vs. Harmlessness One of the most important findings from Constitutional AI research is the trade-off between helpfulness and harmlessness. When you fine-tune a model to be more harmless,to avoid harmful content, to refuse unsafe requests, to be more cautious,you often reduce its helpfulness. The model becomes more likely to refuse benign requests, to give vague answers, or to hedge excessively. This is a fundamental tension. The more constraints you place on a model's behavior, the less useful it becomes in some situations. A model that refuses every request is harmless but useless. A model that helps with everything is useful but potentially dangerous. The goal is to find the right balance,a point on the Pareto frontier where you're getting the maximum harmlessness for a given level of helpfulness, or vice versa. The research shows that Constitutional AI can achieve a better trade-off than traditional RLHF. By carefully tuning the Constitution and the training process, you can get significant improvements in harmlessness with only modest reductions in helpfulness. ### Stage 2: RL with AI Preference Models The second stage of Constitutional AI uses reinforcement learning with AI-generated preference data. Instead of having humans compare pairs of responses, the system uses the Constitution to generate AI preferences: 1. For pairs of responses, the model (guided by the Constitution) decides which one is more aligned,more thoughtful, more respectful, more harmless. 2. These AI-labeled preference pairs are used to train a preference model. 3. The preference model is then used as a reward model for RL fine-tuning. This is structurally similar to RLHF, but the key difference is the source of the preferences. In RLHF, preferences come from human annotators. In Constitutional AI, preferences come from the AI itself, guided by the Constitution. A small amount of human validation data may be used to ensure the AI preferences align with human judgments, but this is a tiny fraction of the data required for full RLHF. ### The Role of Chain-of-Thought One of the most interesting findings is that chain-of-thought reasoning during the critique process significantly improves outcomes. When the model is asked to critique a response, it's not enough to just ask "Is this response harmful?" The model needs to reason through its critique, explaining why the response is or isn't harmful. This chain-of-thought reasoning makes the critique more accurate and more useful for guiding revisions. The same applies to the preference model. When the model is asked to compare two responses, it should reason through the comparison, considering each principle in the Constitution and how each response adheres to or violates it. ### Updating the Constitution One of the most practical advantages of Constitutional AI is that the Constitution can be updated. Social norms change. Laws change. Organizational policies change. When they do, you don't need to collect new human preference data,you just update the Constitution and run additional post-training cycles. This is much faster and cheaper than traditional RLHF, where changing behavioral requirements might require collecting tens of thousands of new human labels. However, there's an open problem: continual learning. How do you update a model's behavior without causing it to "forget" other things it has learned? How do you de-emphasize outdated rules without breaking the model's overall alignment? These are active research questions. ### Validation: Keeping AI Feedback Honest There's a risk with AI-generated feedback: the model might drift. If the AI preference model learns to prefer responses that are different from what humans actually want, the entire training process becomes misaligned. To mitigate this, Constitutional AI uses a human-evaluated validation set. A small set of responses is evaluated by humans, and the AI preference model's judgments are compared to human judgments. If the AI preferences diverge from human preferences, the system is adjusted. This validation set is much smaller than what would be required for full RLHF, but it provides a crucial check on the AI feedback quality. ### Practical Applications of Constitutional AI Constitutional AI isn't just a research curiosity,it has practical applications in many domains. **Enterprise AI assistants**: Companies can write constitutions that encode their specific policies,privacy guidelines, compliance requirements, tone guidelines. The AI assistant is then trained to follow these principles without requiring massive human labeling efforts. **Content moderation**: Constitutional principles can be used to train models that identify and filter harmful content, with the ability to update the principles as new types of harmful content emerge. **Educational AI**: A constitution for an educational AI might include principles about age-appropriateness, encouragement, and avoiding discouraging language. **Domain-specific safety**: Medical, legal, and financial AI systems can have constitutions that encode professional ethics and regulatory requirements. ## Part 5: The Integrated View,Building Self-Improving Agents ### Common Threads Now that we've explored all three paradigms, let's step back and look at what they have in common. All three approaches are built on the same fundamental insight: language models need feedback to improve, and the feedback needs to be structured and meaningful. ReAct gets feedback from tools and the environment. RLEF gets feedback from code execution. Constitutional AI gets feedback from principled AI critique. In each case, the feedback is structured,it's not just "you're wrong," but specific information about what went wrong and how to fix it. All three approaches also use iterative loops: - ReAct: Think → Act → Observe → Think → ... - RLEF: Generate → Test → Repair → Test → ... - Constitutional AI: Respond → Critique → Revise → Respond → ... And all three approaches use language as the medium for reasoning. Thoughts in ReAct are natural language reasoning steps. Error messages in RLEF are text that the model reads and interprets. Critiques in Constitutional AI are natural language explanations of what went wrong. ### How They Work Together These approaches aren't mutually exclusive,they're complementary. In a complex agent system, you might use all three. Consider a coding assistant: - It uses ReAct-style workflows to plan its approach, search for relevant documentation, and decide what to do. - It uses execution feedback (RLEF-style) to test its code, see what fails, and iterate. - It uses constitutional principles to ensure its code follows best practices, avoids security vulnerabilities, and respects user privacy. Or consider a general-purpose AI assistant: - It uses ReAct to search for information, access tools, and ground its responses in real data. - It uses execution feedback when it needs to write or debug code. - It uses constitutional principles to ensure its responses are helpful, harmless, and aligned with user values. ### The Feedback Source Defines the Improvement Pathway Here's the key insight: the source and structure of feedback determine what the agent learns and how well it improves. If you only have environmental feedback (ReAct), your agent learns to navigate tools and gather information, but it doesn't learn to verify its own outputs. If you only have execution feedback (RLEF), your agent learns to write correct code, but it can't reason about the world or access external information. If you only have principled AI feedback (Constitutional AI), your agent learns to align with values, but it can't check facts or test its solutions. The most capable agents combine multiple feedback channels, each providing a different type of learning signal. ### Interpretability as a Design Principle One of the most valuable aspects of these approaches is interpretability. ReAct produces thought-action-observation traces that humans can read and audit. Constitutional AI produces critique traces that explain why a response was revised. Even RLEF's error messages provide insight into what went wrong. This interpretability isn't just a nice-to-have,it's essential for building trustworthy systems. When something goes wrong, you can look at the trace and understand what happened. When you need to comply with regulations, you can demonstrate that your system follows principled reasoning. When you need to debug a system, you can see exactly where it made a mistake. ### Practical Recommendations for Building Self-Improving Agents Let me close this section with some practical recommendations based on everything we've covered. **Start with a small, well-defined action space.** Whether you're building a ReAct-style agent or a coding assistant, don't give your system too many tools. Each tool needs to be well-documented, and the model needs to learn when to use it. **Design feedback loops from the beginning.** Don't build a system that generates outputs without any feedback mechanism. Think about what feedback is available,tools, tests, user responses,and how you can structure it for the model. **Separate training feedback from inference feedback.** The public/private test split in RLEF is a great example. Use fast, granular feedback during inference to guide local improvements, and use held-out feedback during training to ensure genuine generalization. **Write down your principles.** If you're building an AI system that interacts with users, write a constitution that defines what behavior is acceptable and what isn't. This doesn't need to be elaborate,even 10-20 principles can make a difference. **Log everything and audit regularly.** The traces produced by these systems are valuable for debugging, improvement, and compliance. Make sure you're capturing them and reviewing them. **Plan for noisy feedback.** Tools fail. Tests are incomplete. AI critiques are imperfect. Build robustness mechanisms,reflection steps, repeated retrieval, majority voting,into your system. **Be aware of trade-offs.** There's no free lunch. More harmlessness might mean less helpfulness. More grounding might mean higher latency. More feedback might mean more complexity. Understand the trade-offs and choose the right operating point for your application. ## Conclusion: The Path Forward We started this course with a fundamental problem: language models, for all their brilliance, are static. They generate text based on patterns in their training data, with no way to check their outputs, no way to learn from experience, and no way to improve over time. The three paradigms we've explored offer a path forward. ReAct shows us that reasoning and action are complementary. By interleaving explicit thoughts with tool calls and observations, we can ground language models in the real world, reduce hallucinations, and make their decision-making transparent and auditable. The cost is higher inference latency and the need to carefully design action spaces, but the benefits are substantial. RLEF shows us that execution feedback is a powerful learning signal. By actually running code and using the results,passes, failures, timeouts, errors,as feedback, we can train code generation models to produce correct, efficient solutions. The two-tier test strategy prevents overfitting, and the iterative refinement loop enables targeted repairs. The approach has limitations,binary rewards are coarse, and scaling to large codebases is challenging,but it demonstrates the power of grounding AI behavior in concrete outcomes. Constitutional AI shows us that aligned behavior can be learned from principled AI feedback rather than massive human labeling efforts. By writing down principles and using them to guide self-critique and revision, we can train models to be both helpful and harmless. The trade-off between these two goals is real, but Constitutional AI achieves a better balance than traditional RLHF at a fraction of the human cost. Together, these approaches represent a shift from static language models to adaptive agents. Agents that can think and act. Agents that can test their own outputs and learn from failure. Agents that can align their behavior with explicit principles. Agents that improve over time. The field is still young. There are open questions about how to handle noisy environments, how to scale execution feedback to large codebases, how to update constitutions without breaking prior learning, and how to balance competing objectives. But the direction is clear. If you're building AI systems, whether you're a developer, a product manager, or a researcher, these principles should inform your design. Think about feedback loops. Think about grounding. Think about alignment. The most capable AI systems won't be the ones with the most parameters,they'll be the ones that can learn from their own interactions, tools, and values. The future of AI isn't static models generating static text. It's adaptive agents that think, act, observe, and improve. And the techniques we've explored in this course,ReAct, RLEF, and Constitutional AI,are the foundation of that future.

Frequently Asked Questions

What is this FAQ and who is it for?

This FAQ answers the most common questions about building self-improving AI agents through feedback from tools, code execution, and AI-based critique. It covers three core techniques,ReAct, RLEF, and Constitutional AI,and explains how they work, why they matter, and how they fit together. Questions progress from foundational concepts to advanced implementation details, making this a useful reference for business leaders, product managers, and technical practitioners.

Each answer is written to be clear and practical, with real-world context where appropriate. The goal is to help you understand not just what these techniques are, but how they can be applied to actual business problems and product decisions.

Fundamental Concepts

What are language models fundamentally missing, and how do we fix it?

Large language models are exceptionally good at solving natural language processing problems, but they are limited by a lack of grounding in the real world. On their own, they have no access to real-time information, cannot query databases, and cannot execute functions or code based on their outputs. They are essentially sophisticated "thinking" machines without the ability to "act." To make them useful for real-world tasks, we need to equip them with the ability to interact with external environments, tools, and codebases, and to learn from the feedback they receive from those interactions.

What are the core concepts behind "ReAct," "RLEF," and "Constitutional AI"?

These three approaches represent distinct paradigms for self-improvement in AI agents:

ReAct (Reasoning + Acting): This technique combines the internal knowledge of a large language model with the ability to interact with external tools. It interleaves the generation of reasoning traces with the execution of actions. When an observation is returned, the model uses it to inform its next reasoning step.

RLEF (Reinforcement Learning with Execution Feedback): This is a specialized technique used primarily for coding agents. It generates code, runs it against a set of public unit tests, and uses the errors or timeouts generated as feedback. This feedback is then fed back into the model to iteratively refine the code until it passes the tests, using reinforcement learning algorithms like PPO.

Constitutional AI: This technique aims to align AI behavior without relying on massive amounts of human feedback. It uses a set of human-written principles called a "Constitution" to critique and revise model responses. The model uses these principles to generate data, which is then used to fine-tune the base model.

How does grounding work in these AI agents?

Grounding refers to connecting the model's abstract reasoning to something verifiable in the real world. In these frameworks, grounding happens differently for each:

In ReAct, grounding happens through tool calls, such as a search engine lookup. The model's output is tied to verifiable facts or the state of a system it queries.

In RLEF, grounding is achieved through code execution,the model sees the actual result (pass/fail) of its generated code against real tests.

In Constitutional AI, grounding happens through alignment with a defined set of rules. The outputs are aligned with the desired human values outlined in the constitution, grounding it in a specific ethical framework.