ELPO: Ensemble Learning-Based Prompt Optimization for LLMs

Forum topic · QianXun · 2025-11-24

Summary

ELPO (Ensemble Learning Based Prompt Optimization) is an automatic prompt optimization (APO) framework designed to overcome the fragility of single-algorithm approaches and the inefficient management of candidate prompt pools. It combines three core components: a shared multi-generator strategy (including a novel Hard-Case Tracking method that analyzes repeatedly failing samples alongside the prompts that caused those failures), efficient search algorithms (notably the first combination of Bayesian optimization with Multi-Armed Bandit methods in APO, mapping prompts into a continuous high-dimensional embedding space), and an ensemble voting mechanism that aggregates diverse high-quality candidates into a robust final prompt. Experiments show ELPO consistently outperforms state-of-the-art baselines such as APE, PromptAgent, EvoPrompt, and ProTeGi across classification, generation, and multiple-choice tasks, including a 7.6-point F1 improvement on the ArSarcasm sarcasm detection dataset and strong results on LIAR, BBH-navigate, ETHOS, WSC, and GSM8K. Because it operates purely through prompt-response interactions, ELPO works with black-box, API-only LLMs, reduces wasted computation on weak candidates, and produces prompts with better generalization and robustness to unseen data.

Overview

ELPO (Ensemble Learning Based Prompt Optimization) addresses two fundamental weaknesses of existing automatic prompt optimization (APO) methods: the brittleness of relying on a single optimization algorithm, and inefficient use of the candidate prompt pool. Motivated by the No Free Lunch theorem and the observation that individual APO methods are "good but unstable," ELPO treats diverse optimization strategies as ensemble members and aggregates them via voting — the first systematic application of ensemble learning ideas to APO.

Key points

Three-part framework: a shared multi-generator prompt strategy, diversified search algorithms, and a final ensemble voting mechanism.
Hard-Case Tracking: a novel generation strategy that pairs repeatedly misclassified samples with the prompts that caused the failures, prompting the LLM to diagnose root causes and produce fundamentally improved, more generalizable prompts — rather than superficial one-off fixes.
Complementary generators: Bad-Case Reflection (self-critique plus few-shot examples from failures) and Evolutionary Reflection (direct mutation and zero-order generation, inspired by genetic algorithms) balance depth of analysis with breadth of exploration.
Efficient search:
Bayesian optimization using Gaussian Process regression over a continuous high-dimensional embedding space of prompts, with Expected Improvement as the acquisition function — enabling "evaluate a few, infer the whole space" efficiency.
Multi-Armed Bandit (first combined with Bayesian search in APO): prompts are clustered (e.g., K-means) into "arms," and Upper Confidence Bound (UCB) allocates evaluation budgets across clusters.
Ensemble voting: high-performing, structurally diverse candidates from all generators and searchers are aggregated, acting as a "prompt optimization committee" that suppresses single-strategy bias and reduces performance variance.

Experimental results

Consistently beats state-of-the-art APO methods across classification, generation, and multiple-choice tasks.
+7.6 F1 over prior best on ArSarcasm (Arabic sarcasm detection); strong gains also on LIAR (lie detection) and BBH-navigate (navigation reasoning).
Evaluation datasets span task types:

| Dataset | Task | Challenge | | --- | --- | --- | | ArSarcasm | Classification | Arabic sarcasm detection | | LIAR | Classification | Deception detection | | BBH-navigate | Multiple-choice | Multi-step logical instruction following | | ETHOS | Classification | Hate speech detection | | WSC | Multiple-choice | Coreference / commonsense reasoning | | GSM8K | Generative QA | Grade-school math problem solving |

Baselines compared include APE, PromptAgent (MCTS), EvoPrompt/PromptBreeder (evolutionary), ProTeGi (textual gradients), and RLPrompt/TEMPERA (reinforcement learning).
Ablation studies confirm each component's contribution, and show that removing ensemble voting increases performance variance.

Why it matters

Black-box friendly: operates purely on prompt–response interactions, so it works with closed-source, API-only LLMs where gradient-based soft prompt methods are inapplicable.
Cost-efficient: Bayesian + MAB search minimizes expensive LLM evaluations and avoids wasting compute on hopeless prompt variants.
Robust and generalizable: Hard-Case Tracking plus ensemble voting yields prompts that generalize beyond training data, addressing the "myopic" overfitting of feedback-only methods.