3. Automated Self-Healing with LLM-as-a-Judge
3.1. System Architecture: A Fully Programmatic Feedback Loop
This section of the cookbook introduces a fully automated, programmatic approach to the self-evolving loop, eliminating the need for any user interface. This API-driven workflow is designed for scalability and is well-suited for integration into production pipelines and continuous integration/continuous deployment (CI/CD) environments. The architecture of this automated system is centered around a set of Python scripts that orchestrate the entire feedback loop, from generating summaries with the agent to evaluating them with a suite of graders and updating the agent's prompt based on the results. This programmatic approach enables the system to process a large volume of data without requiring manual intervention, making it ideal for continuous monitoring and improvement of agent performance in a production setting. The system is built on top of the OpenAI API, which provides the necessary tools for both generating text with the agent and creating and running evaluations with the graders.
The core components of the automated system are the summarization agent, the metaprompt agent, the evaluation suite, and the orchestration logic. The summarization agent is the primary agent that performs the task of summarizing the regulatory document sections. The metaprompt agent is a separate agent that is responsible for optimizing the summarization agent's prompt based on the feedback from the graders. The evaluation suite is a collection of four distinct graders that assess the quality of the summaries from different perspectives. The orchestration logic is a set of Python functions that tie all of these components together, managing the flow of data between them and controlling the iterative optimization process. This modular architecture makes the system flexible and extensible, allowing for new graders to be added or for the optimization logic to be modified as needed. The following subsections will provide a detailed description of each of these components, illustrating how they work together to create a fully automated, self-healing agent.
3.2. Building the Evaluation Suite: A Multi-Grader Approach
A critical component of the automated self-healing system is the evaluation suite, which is responsible for providing the objective, quantitative feedback that drives the optimization process. This cookbook defines a multi-grader approach, using four complementary graders that balance deterministic checks with semantic judgment. Each grader is designed to assess a specific aspect of the summary's quality, and together they provide a comprehensive and robust evaluation of the agent's performance. The use of multiple graders is a key design choice, as it ensures that the evaluation is not biased towards a single metric and that it captures a wide range of potential failure modes. The scores from each grader are combined into an aggregated score, which is then used to determine whether the agent's performance is acceptable and to guide the prompt optimization process. The following subsections will provide a detailed description of each of the four graders, explaining their purpose, their implementation, and their role in the overall evaluation strategy.
| Grader | Type | Pass Threshold | What It Checks | Why It's Important |
|---|---|---|---|---|
| Chemical Name Preservation | Python | 0.8 | Ensures all exact chemical names from the source text appear in the summary. | Forces preservation of critical domain entities, ensuring scientific and regulatory accuracy. |
| Summary Length Adherence | Python | 0.85 | Measures deviation from a target 100-word length. | Keeps summaries concise and comparable, preventing verbosity from masking poor content. |
| Semantic Similarity | Cosine Similarity | 0.85 | Calculates the cosine similarity between the source text and the summary. | Ensures the summary stays semantically anchored to the source, preventing drift or hallucination. |
| Holistic Quality Assessment | LLM-as-a-Judge | 0.85 | Provides a rubric-driven score from a model acting as an evaluator. | Captures nuanced quality signals that rule-based metrics miss, improving overall robustness. |
#### 3.2.1. Grader 1: Chemical Name Preservation (Python)
The first grader in the evaluation suite is the Chemical Name Preservation Grader. This grader is implemented as a Python function and is designed to ensure that the agent's summary accurately preserves all of the chemical names that appear in the source text. This is a critical requirement for the regulatory document summarization task, as the precise and accurate use of chemical nomenclature is essential for scientific and regulatory clarity. The grader works by first defining a master list of chemical names that are relevant to the dataset. This list includes a wide range of chemical compounds, from the active drug substance to various reagents and solvents used in the manufacturing process. The grader then scans the source section to identify which of these chemical names are present. Finally, it checks the generated summary to see if all of the identified chemical names from the source are also present in the summary. The grader returns a score between 0 and 1, representing the proportion of chemical names from the source that were correctly preserved in the summary.
The implementation of this grader is a good example of how deterministic, rule-based checks can be used to enforce specific domain constraints. The grader's logic is simple and transparent, making it easy to understand and debug. The pass threshold for this grader is set to 0.8, meaning that at least 80% of the chemical names from the source must be present in the summary for the grader to pass. This threshold can be adjusted based on the specific requirements of the task. The grader's focus on a single, well-defined aspect of the summary's quality makes it a powerful tool for ensuring the factual accuracy of the agent's outputs. By including this grader in the evaluation suite, the system can quickly identify and flag any summaries that fail to meet this critical requirement, providing a clear and actionable signal for the prompt optimization process. The grader's output is a key component of the aggregated score, and its failure is a strong indicator that the agent's instructions need to be refined to better emphasize the importance of preserving specific domain entities.
#### 3.2.2. Grader 2: Summary Length Adherence (Python)
The second grader in the evaluation suite is the Summary Length Adherence Grader. This grader is also implemented as a Python function and is designed to ensure that the agent's summaries are concise and adhere to a specified length constraint. For the regulatory document use case, the target length for the summaries is set to 100 words. The grader calculates the word count of the generated summary and then computes the relative deviation from the target length. The grader's scoring function is designed to be lenient, allowing for a 20% tolerance band around the target length. If the summary's length falls within this tolerance band, the grader returns a perfect score of 1.0. If the length falls outside of this band, the score decays linearly, with the score decreasing as the deviation from the target length increases. This approach ensures that summaries that are close to the target length are not penalized too harshly, while still discouraging excessively long or short summaries.
The purpose of this grader is to enforce a stylistic constraint on the agent's outputs, ensuring that the summaries are not only accurate but also well-structured and easy to read. In the context of regulatory documents, where clarity and conciseness are highly valued, this is an important quality to measure. The grader's pass threshold is set to 0.85, which is a relatively high bar, reflecting the importance of this constraint. By including this grader in the evaluation suite, the system can ensure that the agent is not only capturing the necessary information but is also presenting it in a clear and concise manner. The feedback from this grader can be used to guide the prompt optimization process, encouraging the metaprompt agent to generate instructions that emphasize brevity and clarity. The grader's simple, deterministic implementation makes it a reliable and efficient tool for measuring this important aspect of the summary's quality.
#### 3.2.3. Grader 3: Semantic Similarity to Source (Cosine Similarity)
The third grader in the evaluation suite is the Semantic Similarity Grader. This grader uses a text similarity metric, specifically cosine similarity, to measure the semantic overlap between the source section and the generated summary. The purpose of this grader is to ensure that the summary is not just a collection of keywords but is a faithful and accurate representation of the source text's meaning. The grader works by converting both the source text and the summary into high-dimensional vector representations, often referred to as embeddings. The cosine similarity between these two vectors is then calculated, which provides a measure of their semantic similarity. A score of 1.0 indicates that the two texts are semantically identical, while a score of 0.0 indicates that they are completely dissimilar. The pass threshold for this grader is set to 0.85, which is a relatively high bar, reflecting the importance of semantic fidelity in the summarization task.
The use of a semantic similarity metric is a key component of the evaluation suite, as it provides a way to measure the quality of the summary that goes beyond simple keyword matching or length constraints. This grader is particularly effective at catching summaries that are superficially well-formed but that have drifted away from the core meaning of the source text. For example, a summary that paraphrases the source text in a way that introduces subtle inaccuracies or changes in meaning would be penalized by this grader. By including this grader in the evaluation suite, the system can ensure that the agent is not just extracting information from the source text but is also understanding its meaning and preserving it in the summary. The feedback from this grader can be used to guide the prompt optimization process, encouraging the metaprompt agent to generate instructions that emphasize the importance of semantic accuracy and faithfulness to the source.
#### 3.2.4. Grader 4: Holistic Quality Assessment (LLM-as-a-Judge)
The fourth and final grader in the evaluation suite is the Holistic Quality Assessment Grader, which is implemented using an LLM-as-a-judge. This grader is designed to provide a comprehensive, qualitative assessment of the summary's quality, capturing the nuanced signals that the other, more deterministic graders might miss. The grader works by sending the source section and the generated summary to a separate, powerful LLM, which is instructed to act as an expert technical summarization evaluator. The LLM is provided with a detailed rubric that guides its evaluation, asking it to assess the summary's comprehensiveness, faithfulness, and technical accuracy. The rubric provides a clear scoring guideline, with scores ranging from 0 to 1, and detailed descriptions of what each score level represents. The LLM is instructed to respond with only a single number, representing its overall assessment of the summary's quality.
The use of an LLM-as-a-judge is a powerful technique for evaluating the quality of text generation, as it allows for a level of nuance and understanding that is difficult to achieve with rule-based metrics. This grader can assess aspects of the summary's quality that are hard to quantify, such as the clarity of the writing, the logical flow of the information, and the overall effectiveness of the summary in conveying the key points of the source text. The pass threshold for this grader is set to 0.85, which is consistent with the other graders in the suite. The feedback from this grader, which can include a textual rationale for its score, is a valuable input for the prompt optimization process. It provides a high-level, holistic assessment of the agent's performance, which can be used to guide the metaprompt agent in generating more effective and well-rounded instructions. This grader serves as a final failsafe, ensuring that the overall quality of the agent's outputs is high, even if they pass all of the other, more specific checks.
3.3. Orchestrating the Self-Evolving Loop
The orchestration of the self-evolving loop is the process of bringing together all of the individual components—the summarization agent, the metaprompt agent, the evaluation suite, and the versioning system—and coordinating their actions to create a seamless, automated workflow. This is achieved through a set of Python functions that manage the flow of data between the components and control the iterative optimization process. The orchestration logic is responsible for a number of key tasks, including managing the versions of the prompts, translating the feedback from the graders into actionable instructions for the metaprompt agent, and deciding when to promote a new version of the agent to become the new baseline. This orchestration is the glue that holds the entire system together, and it is what enables the creation of a truly autonomous, self-healing agent. The following subsections will provide a detailed description of the key aspects of the orchestration logic, illustrating how the different components are integrated to create a cohesive and effective system.
The orchestration logic is designed to be robust and resilient, with built-in mechanisms for handling failures and ensuring that the system does not get stuck in an infinite loop of unsuccessful optimization attempts. For example, the system keeps track of the best-performing prompt candidate and can revert to it if a new optimization attempt fails to improve performance. The orchestration logic also includes features for observability, such as detailed logging and tracing, which provide a clear view of the system's operation and make it easier to debug any issues that may arise. The modular design of the orchestration logic makes it easy to extend and customize the system to meet the specific needs of a given use case. For example, new graders can be added to the evaluation suite, or the logic for selecting the best prompt candidate can be modified to use a different scoring function. This flexibility is a key advantage of the framework, as it allows it to be adapted to a wide range of different tasks and domains.
#### 3.3.1. Agent and Prompt Versioning for Traceability
A critical aspect of the orchestration logic is the management of agent and prompt versions. In a system that is constantly evolving, it is essential to have a clear and reliable way to track the changes that are made to the agent's instructions. This is important for a number of reasons, including traceability, reproducibility, and the ability to roll back to a previous version in case of a regression. The cookbook introduces a set of Python classes, PromptVersionEntry and VersionedPrompt, to handle this task. The PromptVersionEntry class is a data model that represents a single version of a prompt, including the prompt text, the model version, and any associated metadata. The VersionedPrompt class is a utility that manages a collection of these prompt versions, providing methods for adding new versions, retrieving the current version, and reverting to a previous version.
The use of a formal versioning system is a key design choice that ensures the robustness and reliability of the self-evolving loop. By keeping a complete history of all prompt versions, the system can provide a clear audit trail of the agent's evolution, which is essential for debugging and for understanding the impact of changes. The ability to revert to a previous version is also a crucial safety feature, as it provides a way to recover from a failed optimization attempt or from a new prompt that introduces unexpected problems. The versioning system is integrated into the orchestration logic, with each new optimization attempt resulting in a new version of the prompt being created and added to the history. The system also keeps track of the performance of each prompt version, which allows it to make an informed decision about which version to promote to become the new baseline. This systematic approach to versioning is a key enabler of the system's ability to learn and improve over time, while also ensuring its stability and reliability in a production environment.
#### 3.3.2. The Metaprompt Agent: Translating Feedback into New Instructions
The metaprompt agent is a key component of the automated self-healing system. It is a separate LLM agent whose sole purpose is to act as a prompt optimizer, translating the structured feedback from the graders into a new, improved set of instructions for the summarization agent. The metaprompt agent is guided by a detailed template, METAPROMPT_TEMPLATE, which provides it with the context it needs to perform this task. The template includes the original prompt, the source section, the generated summary, and the consolidated feedback from the graders. The template instructs the metaprompt agent to generate a new prompt that is more specific, more directive, and better aligned with the desired performance criteria. The output of the metaprompt agent is a new, improved prompt that is then used to create a new version of the summarization agent.
The metaprompt agent is a powerful tool for automating the prompt optimization process. It allows the system to explore a wide range of prompt variations without requiring manual intervention, making it ideal for continuous integration and deployment. The use of a separate agent for this task is a key design choice, as it allows for a clear separation of concerns between the task-performing agent and the optimization agent. This makes the system more modular and easier to maintain. The metaprompt agent's ability to interpret the feedback from the graders and translate it into actionable instructions is a key enabler of the system's ability to learn and improve over time. By automating the process of prompt optimization, the metaprompt agent frees up human experts to focus on more strategic tasks, such as defining the overall goals of the system and reviewing its performance at a high level. The metaprompt agent is a crucial link in the self-evolving loop, bridging the gap between the evaluation of the agent's performance and the generation of a new, improved agent.
#### 3.3.3. The Optimization Loop: Evaluating, Scoring, and Updating
The core of the orchestration logic is the optimization loop itself, which is the process of repeatedly evaluating the agent's performance, scoring it against the defined criteria, and updating the prompt based on the feedback. This loop is implemented as an asynchronous Python function, self_evolving_loop, which simulates a stream of incoming requests for summarization by iterating over the rows of the dataset. For each section of the document, the loop performs a series of steps. First, it uses the current version of the summarization agent to generate a summary. Then, it calls the evaluation suite to get the scores from the four graders. The scores are then aggregated, and the loop checks if the performance meets the lenient pass criteria. If it does, the loop moves on to the next section. If it does not, the loop calls the metaprompt agent to generate a new, improved prompt, and the process is repeated for a maximum number of retries.
The optimization loop is designed to be robust and efficient. It uses caching to avoid redundant calls to the evaluation suite for the same section-summary pair. It also keeps track of the best-performing prompt candidate, so that it can be promoted to become the new baseline at the end of the loop. The loop's logic for selecting the best prompt is based on the cumulative performance across all sections, which helps to ensure that the final prompt is the strongest performer overall. The loop also includes detailed logging and print statements, which provide a clear view of its operation and make it easier to debug any issues that may arise. The optimization loop is the engine of the self-evolving system, driving the iterative process of evaluation and refinement that enables the agent to learn and improve over time. It is a complex piece of logic, but it is designed to be modular and easy to understand, with each step of the process clearly defined and separated from the others.
3.4. Observability and Monitoring
In a production environment, it is essential to have a clear and comprehensive view of the operation of the self-evolving system. This is where observability and monitoring come into play. The cookbook demonstrates two key approaches to observability: dashboard tracing and continuous monitoring. Dashboard tracing provides a real-time, visual representation of the workflow and the individual agent calls, making it easy to see how the system is performing and to identify any bottlenecks or errors. Continuous monitoring, on the other hand, involves setting up a scheduled process to periodically re-evaluate the agent's performance on new data, ensuring that the agent remains accurate and compliant as the data distribution evolves. These two approaches work together to provide a complete picture of the system's health and performance, enabling operators to proactively identify and address any issues that may arise. The following subsections will provide a more detailed description of these two approaches, illustrating how they can be used to ensure the reliability and effectiveness of the self-evolving system in a production environment.
The importance of observability and monitoring cannot be overstated. In a system that is constantly changing and evolving, it is crucial to have the tools and processes in place to understand what is happening and why. Without proper observability, it would be impossible to debug the system, to understand the impact of changes, or to ensure that the system is meeting its performance goals. The cookbook's focus on observability is a reflection of its practical, production-oriented approach. It recognizes that a successful self-evolving system is not just one that can learn and improve, but also one that can be effectively managed and operated in a real-world environment. The tools and techniques described in this section are essential for achieving this goal, and they provide a solid foundation for building robust, reliable, and observable self-evolving agents.
#### 3.4.1. Dashboard Tracing for Workflow and Agent Calls
The OpenAI dashboard provides a powerful tool for observing the operation of the self-evolving system. The dashboard's tracing feature allows for the visualization of the entire optimization workflow, from the initial call to the summarization agent to the final update of the prompt. The traces provide a detailed, step-by-step view of the process, showing the inputs and outputs of each agent call, the scores from the graders, and the decisions made by the orchestration logic. This level of detail is invaluable for debugging the system, as it allows developers to see exactly what happened at each stage of the process. The traces can also be used to monitor the performance of the system, providing insights into the latency of each step and the overall throughput of the workflow.
The dashboard's tracing feature is particularly useful for understanding the behavior of the individual agents. By drilling down into a specific agent call, developers can see the exact prompt that was used, the model that was called, and the full text of the generated output. This level of detail is essential for understanding why an agent might have produced a particular output and for identifying any issues with the prompt or the model. The traces also provide a clear view of the flow of data between the different agents, making it easy to see how the feedback from the graders is being used to inform the actions of the metaprompt agent. The dashboard's tracing feature is a key component of the system's observability, providing a real-time, visual representation of the self-evolving loop in action. It is an essential tool for anyone who is building or operating a self-evolving agent, and it provides a level of insight that is simply not possible with traditional logging and monitoring tools.
#### 3.4.2. Continuous Monitoring with Scheduled Re-evaluation
In a production environment, the data that an agent processes is often not static. New data is constantly being generated, and the distribution of that data can change over time. This can lead to a phenomenon known as model drift, where the agent's performance degrades as it is exposed to data that is different from what it was trained on. To address this challenge, it is essential to have a process for continuous monitoring and re-evaluation. The cookbook provides a pseudo-code example of how this can be achieved, using a simple scheduler to periodically check for new data and to trigger the evaluation and optimization loop when new data is detected. This approach ensures that the agent remains accurate and compliant as the data distribution evolves, which is a key requirement for maintaining high-quality, real-world performance.
The continuous monitoring process can be implemented using a variety of tools and techniques, such as a cron job, a lightweight scheduler, or a message queue. The basic idea is to have a process that runs in the background, periodically checking for updates in the data source. When new data is detected, the process automatically triggers the self-evolving loop, which then evaluates the agent's performance on the new data and updates the prompt if necessary. This automated approach to monitoring and retraining is a key advantage of the self-evolving framework, as it allows the system to adapt to changing conditions without requiring manual intervention. This is particularly important in a production environment, where the ability to respond quickly to new data is essential for maintaining a high level of service. The cookbook's discussion of continuous monitoring is a reflection of its practical, production-oriented approach, and it provides a clear and actionable guide for implementing this important capability.
4. Advanced Optimization Strategies
4.1. Model Evaluation and Selection
#### 4.1.1. Comparing Model Candidates (e.g., GPT-5, GPT-5-mini)
The self-evolving loop can be extended beyond prompt optimization to include the evaluation and selection of different model candidates. This is a powerful technique for ensuring that the system is using the most effective and efficient model for a given task. The cookbook provides an example of how to implement this by creating a compare_model_candidates function. This function takes the improved prompt from the metaprompt agent and uses it to generate outputs with two or more different models, such as GPT-5 and GPT-5-mini. The outputs from each model are then evaluated using the same suite of graders, and the model that achieves the highest score is selected as the winner. This approach allows the system to automatically find the optimal balance between performance and cost, as different models may have different strengths and weaknesses, as well as different pricing structures.
The process of comparing model candidates is integrated into the main optimization loop. When a prompt fails to meet the performance threshold, the system first uses the metaprompt agent to generate an improved prompt. It then passes this new prompt to the compare_model_candidates function, which evaluates it across the different model candidates. If one of the models achieves a passing score, that model is selected, and the prompt is updated with the new model version. This automated approach to model selection is a key advantage of the framework, as it allows the system to adapt to the specific requirements of the task without requiring manual tuning. It also provides a way to future-proof the system, as new and improved models can be easily added to the list of candidates and automatically evaluated. The cookbook's discussion of model evaluation and selection is a reflection of its comprehensive, production-oriented approach, and it provides a clear and actionable guide for implementing this advanced optimization strategy.
#### 4.1.2. Integrating Model Selection into the Optimization Loop
The integration of model selection into the main optimization loop is a key feature of the advanced self-evolving system. This is achieved by modifying the self_evolving_loop function to call the compare_model_candidates function when a prompt fails to meet the performance threshold. The modified loop, self_evolving_loop_with_model_comparison, first attempts to improve the prompt using the current model. If that fails, it then calls the compare_model_candidates function to see if a different model can achieve a passing score with the improved prompt. This creates a more robust and flexible optimization process, as it allows the system to explore both prompt and model space in search of the best possible performance. The integration is seamless, with the compare_model_candidates function returning the best-performing agent, which is then used for the next iteration of the loop.
This integrated approach to optimization is a significant improvement over a system that only optimizes the prompt. By considering both the prompt and the model, the system can find solutions that would not be possible with prompt optimization alone. For example, a more powerful model might be able to achieve a passing score with a simpler prompt, while a less powerful model might require a more detailed and specific prompt to achieve the same level of performance. The ability to automatically find the right combination of prompt and model is a key advantage of this approach, as it allows the system to be both effective and efficient. The cookbook's code example provides a clear and practical demonstration of how to implement this integration, making it easy for readers to adapt the technique to their own use cases. This advanced optimization strategy is a powerful tool for building high-performing, production-ready agentic systems.
4.2. Prompt Optimization with Genetic-Pareto (GEPA)
#### 4.2.1. The GEPA Framework: Reflective Evolution for Generalization
The Genetic-Pareto (GEPA) framework represents a more advanced and sophisticated approach to prompt optimization. Unlike the static metaprompt agent, which uses a fixed template to generate new prompts, GEPA employs a more dynamic and reflexive process. It samples agent trajectories, reflects on them in natural language, proposes prompt revisions, and evolves the system through iterative feedback loops. This evolutionary approach is designed to find more robust and generalized prompts that perform well across a wide range of inputs. The GEPA method, as described in the paper by Agrawal et al. , offers a compelling blueprint for continuous, self-improving prompt optimization. The framework is designed to avoid overfitting to a specific dataset or set of graders, which can be a risk with simpler optimization methods.
The core of the GEPA framework is its use of a reflection LM to analyze the performance of the agent and to propose improvements. This reflection LM is given the inputs, outputs, and feedback from the evaluation suite, and it is tasked with identifying the root causes of the agent's failures and with proposing specific changes to the prompt to address them. This reflective process is more nuanced and insightful than the simple template-based approach of the metaprompt agent, and it can lead to more significant and lasting improvements in performance. The GEPA framework also uses a training and validation set to ensure that the evolved prompts are not just memorizing the training data but are actually learning to generalize to new, unseen examples. This focus on generalization is a key advantage of the GEPA framework, and it makes it a particularly good choice for use cases where the agent will be exposed to a wide variety of different inputs.
#### 4.2.2. Adapting the Evaluation Suite for GEPA
To use the GEPA framework, the existing evaluation suite needs to be adapted to work with the GEPA API. This is done by creating an adapter class, EvalsBackedSummarizationAdapter, which implements the three required hooks for the GEPA framework: evaluate, get_components_to_update, and make_reflective_dataset. The evaluate hook runs the summarization and grading pipeline for a given set of inputs and a candidate prompt. The get_components_to_update hook specifies which part of the prompt should be evolved by GEPA (in this case, the system_prompt). The make_reflective_dataset hook packages the inputs, outputs, and feedback into a format that can be read by the reflection LM. This adapter acts as a bridge between the existing evaluation suite and the GEPA framework, allowing the two to work together seamlessly.
The creation of this adapter is a straightforward process that involves wrapping the existing evaluation logic in the required GEPA interface. The evaluate hook calls the run_eval and parse_eval_run_output functions to get the scores from the graders, and it then packages these scores, along with the outputs and feedback, into an EvaluationBatch object. The get_components_to_update hook simply returns a list containing the string "system_prompt". The make_reflective_dataset hook creates a list of examples, where each example contains the input, the generated output, and the feedback for a single evaluation. This adapter allows the GEPA framework to use the same robust evaluation suite that was developed for the automated self-healing loop, ensuring that the optimization process is based on a comprehensive and reliable assessment of the agent's performance.
#### 4.2.3. Running GEPA for Robust, Generalized Prompts
Once the adapter is in place, the GEPA framework can be run using the gepa.optimize function. This function takes the seed candidate (the initial prompt), the training and validation sets, and the adapter as input. It then runs the optimization process, which involves repeatedly evaluating candidate prompts, reflecting on the results, and proposing new, improved prompts. The process continues until a maximum number of metric calls is reached or until the performance on the validation set stops improving. The output of the gepa.optimize function is the best-evolved prompt, which can then be used to create a new, improved version of the summarization agent. The GEPA framework is a powerful tool for finding robust, generalized prompts, and it is a significant step up from the simpler optimization methods described earlier in the cookbook.
The GEPA framework's focus on reflection and generalization makes it a particularly good choice for complex, high-stakes use cases where the agent needs to perform well across a wide range of different inputs. The use of a training and validation set helps to prevent overfitting, and the reflective process of the reflection LM can lead to more insightful and effective prompt improvements. The cookbook's example of running GEPA on the regulatory document summarization task demonstrates the power of this approach. The resulting prompt is highly detailed and specific, with a clear focus on preserving the key technical facts and nomenclature from the source text. This level of detail and specificity is a direct result of the GEPA framework's ability to learn from the feedback and to propose targeted improvements to the prompt. The GEPA framework is a valuable addition to the toolkit of anyone who is building self-evolving agents, and it provides a clear path towards creating more robust, reliable, and high-performing systems.
4.3. Comparing the Three Optimization Strategies
The cookbook presents three distinct strategies for prompt optimization, each with its own strengths and weaknesses. The choice of which strategy to use depends on the specific requirements of the use case, including the need for speed, automation, and generalization. The following table provides a summary of the three strategies, highlighting their key characteristics and when they are most appropriate to use.
| Strategy | Approach | Strengths | Weaknesses | Best For |
|---|---|---|---|---|
| OpenAI Platform Optimizer | Manual feedback via UI, automated optimization. | Speed and ease of use. Excellent for rapid prototyping and gathering human insights. | Not scalable or automated. Requires manual effort for feedback and iteration. | Rapid prototyping, human-in-the-loop scenarios, early-stage development. |
| Static Metaprompt Loop | Automated loop with a fixed metaprompt for optimization. | Lightweight automation. Enables fast feedback loops without human intervention. | Risk of overfitting. Limited exploration space due to a static metaprompt. | Development phase, continuous integration, scenarios with clear, defined graders. |
| GEPA | Evolutionary optimization with reflective, language-based updates. | Systematic generalization. Produces robust, high-performing prompts with strong empirical evidence. | More complex and computationally intensive. Requires a training and validation set. | Production environments, high-stakes tasks, scenarios requiring robust generalization. |
#### 4.3.1. OpenAI Platform Optimizer: Speed and Human Feedback
The OpenAI Platform Optimizer is the simplest and most straightforward of the three strategies. It is designed for speed and ease of use, making it an excellent choice for rapid prototyping and for scenarios where human-in-the-loop oversight is a requirement. The platform's visual interface makes it easy to provide structured feedback, and the automated optimization feature can quickly generate a high-quality prompt based on that feedback. The main strength of this approach is its ability to leverage the nuanced judgment of human experts, which is particularly valuable in the early stages of development when the desired behavior of the agent is not yet fully understood. The platform's tight feedback loop also makes it a great tool for exploring different prompt strategies and for building a shared understanding of the desired outcome among stakeholders.
However, the manual nature of this approach is also its main weakness. The reliance on human reviewers makes it less scalable than the other two strategies, and it is not well-suited for production environments where the agent needs to operate autonomously. The process of providing feedback can also be time-consuming, which can be a bottleneck in a fast-paced development environment. In summary, the OpenAI Platform Optimizer is a powerful tool for the early stages of development and for use cases that require a high degree of human oversight. It provides an excellent foundation for understanding the principles of prompt optimization and for building a high-quality baseline agent, which can then be further refined and scaled using the more automated approaches.
#### 4.3.2. Static Metaprompt Loop: Lightweight Automation
The static metaprompt loop is a step up from the manual approach, providing a lightweight and automated solution for prompt optimization. This strategy uses a separate LLM agent, the metaprompt agent, to automatically generate new prompts based on the feedback from the graders. This eliminates the need for manual feedback, making the process more scalable and suitable for integration into a CI/CD pipeline. The main strength of this approach is its ability to enable fast, iterative development without requiring constant human intervention. The use of a fixed metaprompt template makes the process simple and easy to implement, and the modular architecture of the system makes it flexible and extensible.
However, the use of a static metaprompt is also a potential weakness of this approach. The fixed template may limit the exploration space, and the system may be prone to overfitting to the specific feedback from the graders. The evaluation is also performed on a section-by-section basis, which can lead to a prompt that is optimized for individual examples but does not generalize well to the overall dataset. Despite these limitations, the static metaprompt loop is a powerful and practical solution for many use cases. It provides a good balance between automation and simplicity, and it is a significant improvement over a system that relies solely on manual prompt engineering. This approach is a good choice for the development phase and for scenarios where a lightweight, automated solution is needed.
#### 4.3.3. GEPA: Systematic Generalization and Robustness
The GEPA framework is the most advanced and sophisticated of the three strategies. It uses an evolutionary approach with reflective, language-based updates to find robust, generalized prompts that perform well across a wide range of inputs. The main strength of this approach is its focus on generalization, which is achieved through the use of a training and validation set and a more nuanced, reflective optimization process. The GEPA framework is less likely to overfit to a specific dataset, and it can provide clearer empirical evidence of its performance. This makes it a particularly good choice for high-stakes, production environments where the agent needs to be both accurate and reliable.
The main weakness of the GEPA framework is its complexity and computational intensity. The framework is more difficult to implement than the other two strategies, and it requires a larger dataset and more computational resources to run. However, for use cases where performance and robustness are paramount, the benefits of the GEPA framework far outweigh its costs. The framework's ability to find highly effective, generalized prompts makes it a powerful tool for building state-of-the-art agentic systems. In summary, the GEPA framework is the best choice for production environments and for any use case where the highest level of performance and reliability is required. It represents the cutting edge of prompt optimization, and it provides a clear path towards creating truly intelligent and adaptive agentic systems.
5. Appendix: Reusable Artifacts and Examples
5.1. Example Prompts from Each Optimization Method
This section provides concrete examples of the prompts generated by each of the three optimization strategies, starting from the same simple baseline. These examples illustrate the different levels of detail and specificity that can be achieved with each method, and they provide a clear and tangible demonstration of the power of the self-evolving loop.
#### 5.1.1. Initial Baseline Prompt
The starting point for all three optimization methods was a deliberately simple and generic prompt.
You are a summarization assistant.
Given a section of text, produce a summary.
This minimal prompt serves as a blank slate, allowing the optimization process to demonstrate its ability to evolve a system from a basic instruction to a highly specialized and effective tool.
#### 5.1.2. OpenAI Platform Optimizer Output
The prompt generated by the OpenAI Platform Optimizer, based on structured human feedback, is significantly more detailed. It provides clear instructions on formatting, structure, and content, demonstrating the platform's ability to translate human preferences into a well-defined set of instructions.
You are a summarization assistant.
Task: Summarize the provided text concisely and accurately.
Output requirements:
- Output only the summary. Do not add titles, labels (e.g., "Summary:"), prefaces, or commentary.
- Preserve the document's structure. If multiple sections/subsections appear, summarize each one.
- Use a numbered list for sections/subsections (use their numbers/titles when present).
- Under each, use short dash bullets for key points.
- If there is only a single short section, return a brief bullet list or 1-2 concise sentences.
- Split any inline lists into separate bullets.
- Use plain, simple language. Keep bullets tight (ideally one line each). Remove redundancy.
- Include important quantitative details (values, units, conditions) and constraints. Do not invent information.
- Keep formatting simple: plain text, "1." numbering and "-" bullets only. No tables or special markup.
- Retain exact technical terms/notation from the source (e.g., chemical names, isotopic labels).
- If a section is explicitly marked "Not applicable," include that status; otherwise do not add it.
#### 5.1.3. Static Metaprompt Output
The prompt generated by the static metaprompt agent is even more exhaustive, reflecting its focus on capturing every possible detail from the source text. It is highly structured and directive, leaving little room for interpretation.
You are a technical summarization assistant for scientific and regulatory documentation. Your task is to generate a concise, comprehensive, and fully detailed summary of any scientific, technical, or regulatory text provided. Strictly adhere to the following instructions:
---
**1. Complete and Exact Information Inclusion**
- Capture *every* explicit fact, technical value, specification, quantity, measurement, regulatory reference, entity, process, site, and contextual detail verbatim from the source text.
- Do not omit or generalize any explicit information, no matter how minor.
**2. Precise Terminology and Named Entity Retention**
- Reproduce all names of chemicals, drugs, mixtures, buffer components, devices, companies, institutions, regulatory standards, section numbers, and procedural labels *exactly as stated*.
- Report all quantities, measurements, concentrations, ratios, masses, volumes, compositions, pH values, and units precisely as given.
- Do not paraphrase, rename, substitute, or simplify any term or value.
**3. All Procedural Details and Justifications**
- Explicitly include all described procedures, technical processes (e.g., terminal sterilization, aseptic processing), operational constraints, process justifications, compliance requirements, and standards references.
- Clearly state all reasons provided for choosing or omitting particular methods or processes.
**4. Regulatory and Compliance References**
- Accurately cite all regulations, standards (e.g., USP <797>), compliance statements, section numbers, and cross-references as in the original.
- Include all explicit mentions of compliance, applicability, and site location details.
**5. Explicit Statements of Absence, Limitations, and Applicability**
- Clearly state any declarations of absence, inapplicability (“Not applicable”), or limitations exactly as written in the source.
**6. Structural and Organizational Fidelity**
- Precisely reflect the original document’s section and subsection hierarchy, using clear section labels and indentation.
- Present all enumerations, lists, and tabulated data in structured bullet-point or numbered format, organized in accordance with the source document’s arrangement.
**7. No Paraphrasing, Summarizing, or Reinterpretation**
- Do *not* paraphrase, summarize contextually, reinterpret, or alter the meaning or sequence of any content.
- Remove only literal repetitions or redundant phrasing; otherwise, preserve all explicit statements, technical details, and contextual notes.
---
**Summary Output Objective:**
Produce a summary that delivers the full technical, factual, and regulatory content and structure of the original text, reformatted by eliminating only redundant language. The summary must enable audit, regulatory review, or peer reference without loss of any explicit information or terminology from the source.
---
*Apply these instructions rigorously to every provided document section to ensure scientific and regulatory accuracy and completeness.*
#### 5.1.4. GEPA Optimizer Output
The prompt generated by the GEPA optimizer is the most detailed and specific of all. It is highly tailored to the domain of pharmaceutical regulatory documents, with precise instructions on length, format, and content prioritization. This level of specificity is a direct result of GEPA's reflective and evolutionary approach.
You are a domain-aware summarization assistant for technical pharmaceutical texts. Given a “section” of text, produce a concise, single-paragraph summary that preserves key technical facts and exact nomenclature.
Length and format
- Write 1–3 sentences totaling about 45–70 words (target ~60; never exceed 90).
- Use one paragraph; no bullets, headings, tables, or heavy formatting.
Exact names and notation
- Include every chemical name that appears in the section at least once, using the exact original spelling, capitalization, punctuation, isotopic labels, brackets, hyphens, salts, buffer names, and parenthetical qualifiers. Treat distinct case/format variants as distinct names (e.g., [1-13C]pyruvic acid and [1-13C]Pyruvic acid are separate and each must appear once).
- Examples you must preserve verbatim when present: Hyperpolarized Pyruvate (13C) Injection; non-polarized Pyruvate Injection; Pyruvate (13C) Injection; hyperpolarized [1-13C]pyruvate; Mixture of [1-13C]pyruvic acid and 15 mM AH111501 sodium salt; TRIS/EDTA buffer solution; TRIS; NaOH; Na2EDTA; [1-13C]pyruvic acid; AH111501 sodium salt.
- Also preserve exact study identifiers, batch codes, section numbers, regulatory citations, and instrument parameters as written (e.g., GE-101-001, GE-101-003, USP <797>, 3.2.P.5.2.5, FFF106/140-806, FFF106/142-806, 3T MRI, 5 degree RF pulse, TR=3s, 90 degree pulse, 64 averages, TR=10s, 10 μl Gd/ml solution).
Content prioritization (if space is tight)
1) What the section is about (topic/purpose).
2) All named chemical entities and compositions (list all chemical names at least once; include concentrations/amounts if given).
3) Critical process/handling facts (e.g., aseptic processing vs terminal sterilization; ISO classifications; filtration specs; compounding/filling steps; temperatures/times/volumes; storage/administration limits).
4) Container/packaging specifics (e.g., cryovials, “sterile fluid path”).
5) Microbiological/testing/regulatory details (e.g., sterility/pyrogenicity testing timing; USP <797>; state board compliance; site/manufacturer if stated).
6) Overages/single-dose formulas and key quantities.
Numerical fidelity
- Preserve all critical numbers and units exactly (e.g., 1.44 g, 27.7 mg, 15 mM, 18 mL, 1.47 g, two 0.2 μm filters, ISO 7, ISO 5, 38 mL).
- Include testing/analysis parameters when present (e.g., polarization/relaxation time (T1); number of spectra; pulse angles; TR values; MRI location relative to clean room).
Style and compression
- Be neutral and factual; do not infer unstated information.
- Consolidate repeated statements; compress lists with commas/semicolons to save words.
- Mention tables/figures only to convey key data; do not reproduce them.
- If many chemicals are present, ensure each distinct name appears once; group them succinctly.
- Avoid symbols or special formatting not in the source text.
Common domain cues to include when present
- Aseptic processing vs terminal sterilization and the rationale/timing (e.g., “tested for sterility and pyrogenicity subsequent to patient administration”).
- Environmental/processing controls (ISO 7/ISO 5; LAF unit; filtration; filling/weight targets per cryovial).
- Site/regulatory context (e.g., USP <797>; California State Board of Pharmacy; University of California, San Francisco Department of Clinical Pharmacy).
- Study/kit equivalence statements (e.g., equivalence to GE-101-001/GE-101-003 formulations).
- QC/measurement methods (e.g., capacitive threshold at Administration syringe nominal 38 mL).
Self-check before finalizing
- Does the paragraph contain every distinct chemical name exactly as written in the section (including case and notation variants)?
- Is the summary 45–70 words (≤90), in a single paragraph?
- Are the most critical process/regulatory/testing details and all key numbers preserved without unnecessary verbosity?
5.2. Key Configuration Templates and Code Snippets
This section provides key configuration templates and code snippets that can be reused and adapted for different use cases. These artifacts are the building blocks of the self-evolving system, and they provide a practical starting point for anyone who wants to implement the framework in their own environment.
#### 5.2.1. Evaluation Suite Configuration
The following code snippet shows the configuration for the four-grader evaluation suite. This configuration can be adapted to different use cases by modifying the pass thresholds, the target length for the summary length grader, or the rubric for the LLM-as-a-judge.
testing_criteria = [
{
"type": "python",
"name": "chemical_name_grader",
"image_tag": "2025-05-08",
"pass_threshold": 0.8,
"source": r"""def grade(sample: dict, item: dict) -> float:
# ... (grader logic) ...
return correct / len(present)""",
},
{
"type": "python",
"name": "word_length_deviation_grader",
"image_tag": "2025-05-08",
"pass_threshold": 0.85,
"source": r"""def grade(sample: dict, item: dict) -> float:
# ... (grader logic) ...
return max(0.0, score)""",
},
{
"name": "cosine_similarity",
"type": "text_similarity",
"input": "{{ item.summary }}",
"reference": "{{ item.section }}",
"evaluation_metric": "cosine",
"pass_threshold": 0.85,
},
{
"name": "llm_as_judge",
"type": "score_model",
"model": "gpt-4.1",
"input": [
{
"role": "system",
"content": (
"You are an expert technical summarization evaluator. "
# ... (rubric) ...
),
},
{
"role": "user",
"content": (
"Section:\n{{item.section}}\n"
"Summary:\n{{sample.output_text}}"
),
},
],
"range": [0, 1],
"pass_threshold": 0.85,
},
]
#### 5.2.2. Metaprompt Template
The following is the METAPROMPT_TEMPLATE used to guide the metaprompt agent. This template can be customized to provide different instructions to the metaprompt agent, depending on the desired characteristics of the optimized prompt.
METAPROMPT_TEMPLATE = """
# Context:
## Original prompt:
{original_prompt}
## Section:
{section}
## Summary:
{summary}
## Reason to improve the prompt:
{reasoning}
# Task:
Write a new summarization prompt that is significantly improved and more specific than the original.
The new prompt should instruct the model to produce concise yet comprehensive technical summaries that precisely preserve all explicit information from the source text. It should emphasize the inclusion of all named entities, quantities, compounds, and technical terminology without paraphrasing or omission. The resulting prompt should read like a clear, directive system message for a technical summarization assistant—structured, unambiguous, and generalizable across scientific or regulatory document sections.
"""
#### 5.2.3. GEPA Adapter Implementation
The following is a simplified version of the EvalsBackedSummarizationAdapter class, which is used to integrate the evaluation suite with the GEPA framework. This adapter can be adapted to different use cases by modifying the evaluate method to use a different set of graders or a different evaluation logic.
class EvalsBackedSummarizationAdapter:
propose_new_texts = None
def __init__(self, client, eval_id: str, gen_model: str = "gpt-5"):
self.client = client
self.eval_id = eval_id
self.gen_model = gen_model
def _summarize(self, system_prompt: str, section: str) -> str:
# ... (summarization logic) ...
return resp.choices[0].message.content.strip()
def evaluate(self, inputs: list[dict], candidate: dict, capture_traces: bool = True) -> EvaluationBatch:
system_prompt = candidate["system_prompt"]
scores: list[float] = []
outputs: list[str] = []
trajectories: list[dict] = []
for item in inputs:
section = item["content"]
summary = self._summarize(system_prompt, section)
outputs.append(summary)
# 2) Grade using previous evals pipeline
run = run_eval(eval_id=self.eval_id, section=section, summary=summary)
out_items = poll_eval_run(eval_id=self.eval_id, run_id=run.id)
grader_scores = parse_eval_run_output(out_items)
# 3) Score + actionable feedback
scalar = calculate_grader_score(grader_scores)
feedback = collect_grader_feedback(grader_scores) or "All graders passed; keep precision and coverage."
scores.append(float(scalar))
trajectories.append({
"inputs": {"section": section},
"generated_output": summary,
"metrics": {"combined": float(scalar), "by_grader": grader_scores},
"feedback": feedback,
})
return EvaluationBatch(scores=scores, outputs=outputs, trajectories=trajectories)
def get_components_to_update(self, candidate: dict) -> list[str]:
return ["system_prompt"]
def make_reflective_dataset(self, candidate: dict, eval_batch: EvaluationBatch, components_to_update: list[str]) -> dict:
examples = []
for traj in (eval_batch.trajectories or []):
examples.append({
"Inputs": {"section": traj["inputs"]["section"]},
"Generated Outputs": traj["generated_output"],
"Feedback": traj["feedback"],
})
return {"system_prompt": examples}
--- : GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning by Lakshya A Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Arnav Singhvi, Herumb Shandilya, Michael J Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexandros G. Dimakis, Ion Stoica, Dan Klein, Matei Zaharia, Omar Khattab - https://arxiv.org/abs/2507.19457