Free for humans

A benchmark for agents that build a retrieval system from a blank page

Retrieval-augmented generation still takes a lot of engineering because the right design changes with the task. AutoRAGBench asks how far a language-model agent can get when it has to build and revise that system itself.

arXiv:2611.255825 min readScore 88/100 · editorial triage · not peer reviewPaper hub2026-W48

The 30-second take

  • AutoRAGBench has nine tasks where agents build retrieval systems from scratch, with no starter code and no prescribed strategy.
  • Each attempt gets a six-hour budget on a single H800 GPU, and the study compares nine agent configurations.
  • GPT-6.1 Sol with Codex CLI scored 56.1 on average, above a composite human baseline of 44.4, and gained 16.6 points when it started from human designs on seven tasks.

What the paper actually did

The authors introduce AutoRAGBench, a benchmark for how well language-model agents develop retrieval-augmented generation systems from scratch in an open code space. It contains nine development tasks. On each task, the agent gets no starter code and no prescribed strategy. It has a six-hour budget on a single H800 GPU. The agent is expected to analyze the task data, implement a retrieval pipeline, and iteratively refine the design from experimental feedback. The study evaluates nine agent configurations and looks at both the quality of the resulting systems and the paths the agents took while building them. GPT-6.1 Sol with Codex CLI records the highest average score, 56.1. A composite human state-of-the-art baseline averages 44.4.

What makes this disruptive

Retrieval-augmented generation is a settled idea. Building a good system is not. Effective choices vary by task and domain, and the work usually takes repeated engineering experiments. AutoRAGBench moves the question from whether a model can answer with retrieved text to whether an agent can construct the retrieval system itself. The authors present it as the first benchmark that asks for that construction from scratch in an open code space, rather than inside a fixed template. The from-scratch result and the improvement result answer different questions. From scratch, the leading configuration averages 56.1 against a composite human baseline of 44.4. Starting from a human-designed system on each of seven tasks, the same configuration improves the score by 16.6 points on average.

Why it matters (outside the lab)

Teams that ship retrieval systems spend much of their time on pipeline choices, not on the language model alone. If an agent can analyze task data, write the pipeline, and revise it from experimental feedback inside a fixed time and hardware budget, some of that iteration can be automated. The abstract's conclusion is that language-model agents can automate effective development of these systems. The scores also show the limit of that sentence. The leading average is 56.1, and the human composite baseline is 44.4, so the agent lead is real on the benchmark's own averages and is not described as a solved engineering problem. The separate 16.6 point gain matters because it starts from a human design on seven tasks, which is closer to how a team would actually hand work to an agent.

Limitations & open questions

The abstract does not define the score scale, so 56.1 and 44.4 should not be read as percentages or as fractions of a stated maximum. It does not name the nine tasks, the nine agent configurations, or the seven tasks used in the from-human comparison. It does not show the development trajectories in any detail, even though it says those trajectories were analyzed. It does not report cost beyond the six-hour budget on one H800 GPU, and it does not show how the systems behave outside this benchmark. The two numeric comparisons are different setups and are not one improvement figure.

Explain ladder

Default article depth

A retrieval-augmented system looks up external documents and then asks a language model to answer from them. Building a good one still takes trial and error, and the right design depends on the job. AutoRAGBench makes agents do that building with no starter code, for nine tasks, within six hours on one H800 GPU. The best reported configuration, GPT-6.1 Sol with Codex CLI, averages 56.1. A composite human baseline averages 44.4. On seven tasks, starting from an existing human design, that same configuration improves the score by 16.6 points on average.

Key terms

Retrieval-augmented generation
A setup that augments a large language model with information retrieved from an external collection of documents. The paper abbreviates it as RAG and treats the construction of these systems as the engineering problem under test.
Open code space
The agent writes the retrieval system as code, from scratch, rather than filling in a template. The benchmark also withholds starter code and prescribed strategies.
Human-SOTA baseline
The composite human state-of-the-art baseline in this benchmark. The abstract reports its average as 44.4 and does not describe how the composite was assembled.
H800
The graphics processor named as the single device each agent may use. The time budget on that device is six hours per task. The abstract does not give further hardware specifications.

Sources

Related explainers

Same topic and week first — keep exploring the scarcity → abundance map.

Editorial explainer · not peer review · always read the primary paper.

Byline: Disruptive Concepts editorial.