ELPO: Ensemble Learning-Based Prompt Optimization for LLMs

Forum topic · QianXun · 2025-11-24

Summary

ELPO (Ensemble Learning Based Prompt Optimization) is an automatic prompt optimization (APO) framework that addresses two core weaknesses of existing APO methods: the fragility of relying on a single optimization algorithm and inefficient management of candidate prompt pools. ELPO combines a shared multi-generator prompt generation strategy—including a novel Hard-Case Tracking technique that analyzes repeatedly misclassified samples alongside the failing prompts—Bad-Case Reflection, and Evolutionary Reflection (direct mutation and zero-order generation). For search, it pioneers combining Bayesian optimization (mapping prompts into a continuous high-dimensional embedding space with Gaussian process regression and Expected Improvement acquisition) with a Multi-Armed Bandit (MAB) approach using Upper Confidence Bound over clustered prompt groups, minimizing costly LLM API calls. A final ensemble voting mechanism aggregates high-performing but structurally diverse candidate prompts to improve accuracy and robustness. Experiments across classification, generation, and multiple-choice tasks (ArSarcasm, LIAR, ETHOS, BBH-navigate, WSC, GSM8K) show ELPO consistently outperforms state-of-the-art baselines such as APE, PromptAgent, EvoPrompt, and ProTeGi, including a 7.6-point F1 improvement on ArSarcasm sarcasm detection. Ablation studies validate each component's contribution. As a black-box method, ELPO works with API-only closed-source LLMs while reducing compute waste and improving generalization.

Overview

ELPO (Ensemble Learning Based Prompt Optimization) is an automatic prompt optimization (APO) framework designed to overcome two fundamental problems in existing APO methods: the fragility of relying on a single optimization algorithm and inefficient management of the candidate prompt pool. Motivated by the No Free Lunch theorem, ELPO treats individual APO algorithms—which are "good but unstable"—as ideal candidates for ensemble aggregation, and builds an adaptive, scalable prompt engineering framework for LLMs.

Key Points

1. Framework: three core components

Shared generation strategies: a multi-generator ensemble framework that includes:
Hard-Case Tracking (novel): identifies samples that repeatedly fail across iterations, jointly analyzes them with the prompts that caused the failures, and uses the LLM to generate fundamentally more robust and generalizable prompts—rather than superficial "patch" fixes.
Bad-Case Reflection: generates self-reflection prompts, iteratively optimizes the system prompt, and uses failure cases as few-shot examples.
Evolutionary Reflection: uses direct mutation (local refinement) and zero-order generation (creating new prompts from population characteristics) to balance local optimization and global exploration.
Diverse search methods: ELPO is the first work, per the authors, to combine Bayesian Search with Multi-Armed Bandit (MAB) in APO:
Bayesian optimization maps discrete prompts into a continuous high-dimensional embedding space, fits a Gaussian process regression surrogate, and uses Expected Improvement (EI) as the acquisition function—enabling "evaluate a few prompts, infer the whole space" efficiency and far fewer LLM queries.
MAB clusters candidate prompts (e.g., via K-means) into "arms" and allocates evaluation budget with the Upper Confidence Bound (UCB) criterion, complementing Bayesian search at the cluster level.
Ensemble voting mechanism: a candidate pool of high-performing but structurally diverse prompts (from different generators and searches) is aggregated via voting—functioning like a "prompt optimization committee"—to select the most robust final prompt and avoid bias from any single strategy.

2. Goals: accuracy, robustness, generalization

Accuracy: explores a broader search space; e.g., a +7.6 F1 improvement on the ArSarcasm sarcasm detection dataset over prior state-of-the-art.
Robustness: ensemble voting reduces performance variance caused by algorithmic randomness or task sensitivity.
Generalization: Hard-Case Tracking targets root causes of recurring errors, producing prompts that transfer to unseen data.

3. Experimental evaluation

Datasets/tasks span three task types:

Baselines compared against major APO paradigms: APE (generate-and-select search), PromptAgent (Monte Carlo tree search), EvoPrompt/PromptBreeder (evolutionary), ProTeGi (textual gradients / LLM feedback), RLPrompt/TEMPERA (reinforcement learning).
Results: ELPO consistently outperforms state-of-the-art methods across classification, generation, and multiple-choice tasks, with gains in F1, accuracy, precision, and recall. Ablation studies confirm each component's contribution, and show that removing ensemble voting increases performance variance.

4. Positioning vs. prior work

Early APO methods (APE, PromptAgent, GPS, EvoPrompt, PromptBreeder) treat optimization as search/evolution but are inefficient and lack directionality.
Feedback-driven methods (ProTeGi's "textual gradients") are more directed but rely on a single algorithm and are "myopic," discarding historical feedback between iterations.
ELPO's key innovation is systematically importing ensemble learning into APO: aggregating multiple diverse, individually unstable optimizers into a committee that is collectively more accurate and stable.

5. Practical value

Black-box friendly: operates purely via prompt–response interaction, so it works with closed-source, API-only LLMs (no gradients or internal states needed).
Cost-efficient: Bayesian + MAB search concentrates evaluation budget on promising candidates, avoiding wasted API calls on low-potential prompt variants.
Generalizable outputs: voting plus Hard-Case Tracking yields prompts that remain reliable on new, unseen data—an important property for production LLM applications.

Conclusion

ELPO reframes APO from "find the single best optimization algorithm" to "build the best ensemble system." By integrating multi-strategy generation, principled Bayesian/MAB search, and robust ensemble voting, it achieves state-of-the-art accuracy, lower variance, and better generalization across diverse NLP tasks while remaining applicable to black-box LLMs.