Beyond the Single Model: Routing for Diversity in Large Language Model Ensembles

When we ask a language model to generate a creative story, suggest a project title, or brainstorm solutions to an open ended problem, we rarely want a single answer. We want a spectrum of valid, high quality possibilities that capture the breadth of human creativity. Yet the vast majority of LLM evaluation remains fixated on single sample accuracy or perplexity, treating generation as a search for one correct response rather than a mapping of possibility space. In their recent preprint, "No Single Best Model for Diversity: Learning a Router for Sample Diversity," Liu et al. confront this gap directly, introducing both a rigorous metric for evaluating diverse generation and a practical system for achieving it through model routing.

The Measurement Problem: Defining Diversity Coverage

The fundamental challenge in evaluating open ended generation is that quality and diversity often sit in tension. A model can achieve high diversity by producing incoherent or irrelevant outputs, while a model that generates the same high quality response repeatedly achieves perfect quality but zero coverage of the valid answer space. Previous work has approached this through distinct diversity metrics (such as lexical distinctness or semantic embedding distances) and quality metrics (such as automated judges or human ratings), but these treat the two dimensions separately.

Liu et al. propose diversity coverage, a metric that jointly evaluates both properties. Conceptually, diversity coverage measures the total quality scores assigned to each unique answer in a generated set, normalized against the best possible answer set of the same size. If a model generates ten responses, the metric identifies the unique valid answers among them, sums their quality scores, and compares this sum to the theoretical maximum achievable with ten distinct, high quality responses. This formulation penalizes redundancy (duplicate answers contribute nothing additional) while rewarding both semantic diversity and individual response quality.

Using this metric, the authors evaluate 18 different LLMs across several datasets including NB-WildChat (1,000 open ended prompts from real user conversations), NB-Curated (92 hand crafted open ended queries), and Simple Questions (a baseline with fixed answer sets). The results reveal a striking pattern: no single model dominates across the benchmark. While some models perform consistently well on average, the "best" model for diversity coverage varies dramatically by prompt.

The Heterogeneity of Model Behavior

Perhaps the most surprising finding in the study is the degree of complementarity between models. The authors identify that for any given prompt, there typically exists a specific model that significantly outperforms all others at generating a diverse, high quality answer set. If one could select the optimal model for each query, an oracle router would achieve 33.0% diversity coverage on NB-WildChat, compared to 23.8% for the single best overall model. This represents a 38.6% relative improvement that remains untapped when relying on monolithic solutions.

Crucially, this heterogeneity does not correlate with simple heuristics such as model size or architectural family. The authors note that models of different families and parameter counts excel on disjoint subsets of queries. A smaller open weights model might generate more diverse responses to a creative writing prompt, while a larger proprietary model dominates factual brainstorming tasks. This suggests that diversity is not merely a function of model capacity or training data size, but rather an emergent property shaped by fine tuning details, instruction following behavior, and possibly unknown architectural inductive biases.

The implication is profound: the current paradigm of selecting "the best" model for a task and deploying it universally is suboptimal for open ended generation. The diversity coverage gap between oracle selection (33.0%) and single model deployment (23.8%) indicates that we are leaving significant value on the table by not tailoring model selection to individual queries.

Learning to Route: From Oracle to Practice

Given that optimal model selection requires knowing the ground truth diversity coverage for each query, which is impossible at inference time, Liu et al. frame the problem as a supervised classification task. They train a diversity router, a model that predicts which LLM in a candidate pool will achieve the highest diversity coverage for a given input prompt.

The training procedure involves generating diversity coverage scores for each model on a training set of queries, then training the router to map from query embeddings to model selection. On NB-WildChat, this trained router achieves 26.3% diversity coverage, closing roughly half the gap between the single best model baseline (23.8%) and the theoretical oracle (33.0%). The system also generalizes to out of domain data (NB-Curated) and remains robust across different answer generation prompting strategies, suggesting the router learns transferable features of query diversity rather than spurious dataset specific patterns.

The authors further explore ensembling, using the router to select not one but two models per query. This hybrid approach yields additional gains, demonstrating that the router can identify complementary model pairs whose outputs together cover more of the valid answer space than either model alone.

Analysis: Why Routing Works and Where It Fails

The success of the diversity router highlights an underappreciated reality in the current LLM landscape: model heterogeneity is an asset, not a bug. As the field fragments into specialized models, instruction tuned variants, and domain specific systems, the assumption that scale alone determines capability becomes increasingly untenable. For diversity coverage specifically, different training objectives and data mixtures appear to carve out distinct "cognitive styles" in models, making them suited to different types of open ended exploration.

However, the router approach introduces its own complexities. The training of the router requires evaluating multiple models on a representative training set to generate labels, a computationally expensive proposition. Furthermore, the diversity coverage metric itself depends on automated quality judges, which may introduce bias or fail to capture nuanced aspects of response validity. If the quality scorer systematically underestimates certain types of creative outputs, the entire optimization target becomes skewed.

There is also the question of inference cost. While routing to a single model is cheaper than running an ensemble of all candidates, it still requires inference on the router itself plus the selected LLM. The paper does not extensively analyze the latency implications, but in production systems, the overhead of routing decisions must be weighed against the marginal diversity gains.

Future Directions: Confidence, Hybrids, and Open Questions

The gap between the trained router (26.3%) and the oracle (33.0%) suggests substantial room for improvement. One promising direction, hinted at in follow up discussions, involves confidence aware routing. If the router can accurately estimate its own uncertainty, it could fall back to ensemble sampling or multi model generation specifically when confidence is low. If routing errors cluster on specific prompt types, a hybrid strategy that dynamically adjusts the number of models based on query characteristics could push coverage well past 30% without requiring oracle access.

More broadly, this work raises questions about the future architecture of LLM systems. Rather than training ever larger monolithic models, we may see the emergence of model ecosystems where lightweight routers orchestrate collections of specialized generators. This paradigm mirrors mixture of experts architectures but operates at the model level rather than the parameter level, allowing for greater modularity and interpretability.

The study also opens methodological questions. How does diversity coverage correlate with human judgments of usefulness? Does optimizing for diversity coverage improve performance on downstream tasks like synthetic data generation or red teaming? And can we develop routers that optimize for diversity directly rather than predicting diversity scores post hoc?

As we move beyond the era of single sample evaluation, Liu et al.'s work provides both the metrics and the machinery necessary to build systems that truly explore the space of possible answers. The recognition that no single model owns the frontier of creativity, and that intelligent routing can unlock latent capabilities across the model landscape, represents a necessary evolution in how we deploy and evaluate large language models.