ELPO: Ensemble Learning Based Prompt Optimization for LLMs

Forum topic · QianXun · 2025-11-24

Summary

ELPO (Ensemble Learning Based Prompt Optimization) is a framework for automatic prompt optimization (APO) of large language models that combines ensemble learning principles with efficient black-box search. ELPO integrates three core components: shared prompt generation strategies that boost candidate diversity, multiple search algorithms including Bayesian optimization (Gaussian process regression with expected improvement acquisition) and multi-armed bandits (clustering candidate prompts with UCB-guided exploration), and an ensemble voting mechanism to select the final prompt. A key innovation, Hard-Case Tracking, analyzes samples that are persistently misclassified across iterations along with the prompts that caused failures, then uses the LLM to generate more robust replacement prompts. Evaluated on classification, generation, and multiple-choice benchmarks including ArSarcasm, LIAR, BBH-navigate, and GSM8K, ELPO outperforms prior state-of-the-art methods, improving F1 by 7.6 points on ArSarcasm, while reducing the number of LLM API calls. Ablation studies confirm the contribution of each component. ELPO is designed for black-box, API-only LLMs, making it practical for enterprise AI, research, and product development workflows.

ELPO: Ensemble Learning Based Prompt Optimization

ELPO (Ensemble Learning Based Prompt Optimization) is a framework that applies ensemble learning ideas to automatic prompt optimization (APO) for large language models, improving accuracy, robustness, and generalization.

Key points

Performance: ELPO consistently outperforms state-of-the-art APO methods, achieving a +7.6 F1 improvement on the ArSarcasm dataset and leading across multiple benchmarks spanning classification, generation, and multiple-choice tasks.
Core innovations: It combines Hard-Case Tracking, Bayesian optimization, multi-armed bandits (MAB), and an ensemble voting mechanism into a robust prompt optimization pipeline.
Practical design: Built for black-box LLM optimization via API interaction, substantially reducing the number of LLM API calls while maintaining performance.

Methodology

ELPO addresses limitations of traditional APO through three elements: shared generation strategies, diverse search methods, and ensemble voting (arXiv:2511.16122).

Hard-Case Tracking

The core novel strategy focuses on samples that remain misclassified across iterations and the prompts that caused the errors:

1. Identify samples persistently misclassified across multiple iterations. 2. Analyze the failing prompts to understand root causes. 3. Use the LLM to generate more robust, generalizable replacement prompts.

Efficient Search Algorithms

Bayesian optimization: Maps prompts into a continuous high-dimensional space and optimizes with Gaussian process regression and an expected improvement acquisition function. Benefits: fewer LLM API calls, intelligent exploration–exploitation trade-off.
Multi-armed bandits: Candidate prompts are clustered, each cluster treated as an arm, with Upper Confidence Bound (UCB) guiding exploration. This is the first application of MAB to APO, enabling efficient evaluation resource allocation.

Experimental evaluation

| Dataset | Task type | Main challenge | |---|---|---| | ArSarcasm | Text classification | Arabic sarcasm detection | | LIAR | Text classification | Lie detection | | BBH-navigate | Multiple choice | Navigation reasoning | | GSM8K | Generative QA | Math problem solving |

Ablation studies

Removing Hard-Case Tracking causes large performance drops, confirming its key role in generalization.
Replacing Bayesian + MAB search with basic methods reduces efficiency.
Selecting a single prompt (no ensemble voting) increases performance variance, demonstrating the value of the voting mechanism.

Applications

NLP tasks: text classification and sentiment analysis (e.g., sarcasm and hate speech detection), question answering and reading comprehension (e.g., WSC coreference), complex reasoning and math (GSM8K).
Challenges addressed: works on closed-source LLM APIs without internal model access; reduces compute waste; produces prompts that generalize across domains and tasks.
Practical value: enterprise AI prompt optimization, academic research tooling, and faster AI product iteration.

References

Paper: ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models
Review: themoonlight.io review

Conclusion

ELPO demonstrates that ensemble learning—combining diverse generation, efficient search, and voting-based selection—can make prompt optimization more accurate, robust, and cost-effective, especially for black-box LLMs accessed via API.