Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation

The 30-second take

• What: The paper casts multi-model routing with costly quality estimates as Pandora’s Box and derives value-of-information policies (Pandora’s Router and Bidder).
• Why it matters: You can match exhaustive high-quality routing while calling expensive estimators far less often—cutting cost without giving up allocation quality.
• Who should care: Teams running multi-LLM stacks, RAG specialist fleets, or inference-time reasoning menus who need principled cost–quality routing.

What the paper actually did

Heterogeneous AI systems—multiple models, architectures, harnesses, or inference-time settings—can raise quality and efficiency by sending each query to the specialist that answers best at lowest cost. Doing that well requires estimating each specialist’s expected return, and those estimates themselves cost money: cheap embedding-based predictors are fast but noisy; accurate estimators (fine-tuned models with retrieval results or partial reasoning traces) are expensive.

The authors formalize the tradeoff as Pandora’s Box, the classic optimal-search problem with costly inspection. Under a Gaussian signal model, the resulting policies have closed-form value-of-information expressions that decide, for each specialist and input, whether refining the value estimate is worth its cost. The centralized policy is Pandora’s Router.

They also study a decentralized variant, Pandora’s Bidder, where specialists independently choose whether to invest in self-assessment before accepting an offered price to claim a query. Experiments across three domains—a standard multi-LLM benchmark, retrieval-augmented specialists, and LLMs with variable inference-time reasoning—show Pandora’s Router matches exhaustive estimation’s routing quality while querying the expensive estimator far less often. In the decentralized setting, value-of-information reasoning improves allocative efficiency when competing estimates are accurate; when they are noisy, it can raise the strategic specialist’s utility at others’ expense.

What makes this disruptive

Most routing stacks either always use a cheap scorer (and misallocate) or always pay for a heavyweight judge (and burn money). Treating estimation as costly inspection with closed-form VOI thresholds reframes routing as optimal search: inspect only when the information is worth the fee. That breaks the implicit assumption that better routing must mean more estimation.

Why it matters (outside the lab)

Multi-model products are becoming the default way to serve diverse queries. The abundance move is not “one giant model for everything,” but fleets of specialists whose expensive judgment is used sparingly and wisely. Principled routers turn today’s luxury—always calling the best judge or the deepest reasoner—into a selective default that preserves quality while shrinking billable inference.

Limitations & open questions

Closed-form results rely on a Gaussian signal model; behavior under other noise models is not claimed in the abstract. Decentralized Pandora’s Bidder can improve system efficiency when estimates are accurate, but with noisy competing estimates it may help a strategic specialist at others’ expense—so incentive alignment is not automatic. Experiments cover three domains described in the abstract; generalization beyond those settings should be treated as open. The abstract reports matching exhaustive routing quality with fewer expensive queries, not zero estimation cost or perfect allocation.