ELPO: Ensemble Learning Based Prompt Optimization for LLMs

Forum topic · QianXun · 2025-11-24

Summary

ELPO (Ensemble Learning Based Prompt Optimization) is an automatic prompt optimization framework for large language models that integrates multiple generation strategies, diverse search algorithms, and an ensemble voting mechanism to select the final prompt. Its key innovations include Hard-Case Tracking, which analyzes persistently misclassified samples and the prompts causing errors to generate more robust candidates, Bayesian optimization that maps prompts into a continuous high-dimensional space using Gaussian process regression and expected improvement, and a multi-armed bandit approach with UCB applied to clustered candidate prompts. Designed for black-box LLMs accessed via API, ELPO reduces the number of LLM calls while maintaining performance. Experiments show it consistently outperforms state-of-the-art methods, achieving a 7.6-point F1 improvement on the ArSarcasm dataset, with strong results across classification (ArSarcasm, LIAR), multiple-choice reasoning (BBH-navigate), and generative math tasks (GSM8K). Ablation studies confirm the contribution of each component, showing larger performance drops without Hard-Case Tracking, basic search methods, or single-prompt selection.

Key points

ELPO (Ensemble Learning Based Prompt Optimization) is a framework for automatic prompt optimization (APO) of large language models, addressing limitations of traditional APO approaches through three core elements: shared generation strategies, diversified search methods, and an ensemble voting mechanism (arXiv paper).

Core methodology

Multi-generator framework: multiple generation strategies increase the diversity and quality of candidate prompts.
Hard-Case Tracking: ELPO's signature innovation — it identifies samples that remain misclassified across iterations, analyzes the prompts responsible for the errors, and uses the LLM to generate improved, more generalizable prompts.
Bayesian optimization: maps prompts into a continuous high-dimensional space using Gaussian process regression with an expected improvement acquisition function, enabling an intelligent exploration-exploitation trade-off and fewer LLM API calls (PDF).
Multi-armed bandit (MAB): candidate prompts are clustered, each cluster serving as an arm, with the upper confidence bound (UCB) criterion guiding exploration — reportedly the first application of MAB in the APO domain.
Ensemble voting: a robust voting mechanism selects the final prompt from candidates, improving stability.

Performance

ELPO consistently outperforms state-of-the-art APO methods across classification, generation, and multiple-choice tasks.
+7.6 F1 improvement over SOTA on the ArSarcasm (Arabic sarcasm detection) dataset.
Evaluated datasets include:

| Dataset | Task | Challenge | |---|---|---| | ArSarcasm | Text classification | Arabic sarcasm detection | | LIAR | Text classification | Lie detection | | BBH-navigate | Multiple choice | Navigation reasoning | | GSM8K | Generative QA | Math problem solving |

Ablation findings

Removing Hard-Case Tracking causes a significant performance drop, confirming its role in generalization.
Replacing the Bayesian + MAB search with basic methods reduces efficiency.
Selecting a single prompt instead of ensemble voting increases performance variance.

Practical significance

Designed for black-box LLMs: works via API without access to model internals, making it suitable for closed-source models.
Reduces LLM API calls, lowering compute cost.
Applicable to text classification, sentiment analysis, QA/reading comprehension, and complex reasoning/math tasks; relevant for enterprise AI, academic research, and product development.

Conclusion

ELPO demonstrates that ensemble learning — combining diverse prompt generation, efficient search, and voting-based selection — yields prompt optimization that is more accurate, robust, and generalizable than single-strategy approaches, while remaining practical for black-box, API-only LLM deployment.