A benchmark for agents that build a retrieval system from a blank page
Retrieval-augmented generation still takes a lot of engineering because the right design changes with the task. AutoRAGBench asks how far a language-model agent can get when it has to build and revise that system itself.
The 30-second take
- AutoRAGBench has nine tasks where agents build retrieval systems from scratch, with no starter code and no prescribed strategy.
- Each attempt gets a six-hour budget on a single H800 GPU, and the study compares nine agent configurations.
- GPT-6.1 Sol with Codex CLI scored 56.1 on average, above a composite human baseline of 44.4, and gained 16.6 points when it started from human designs on seven tasks.
What the paper actually did
The authors introduce AutoRAGBench, a benchmark for how well language-model agents develop retrieval-augmented generation systems from scratch in an open code space. It contains nine development tasks. On each task, the agent gets no starter code and no prescribed strategy. It has a six-hour budget on a single H800 GPU. The agent is expected to analyze the task data, implement a retrieval pipeline, and iteratively refine the design from experimental feedback. The study evaluates nine agent configurations and looks at both the quality of the resulting systems and the paths the agents took while building them. GPT-6.1 Sol with Codex CLI records the highest average score, 56.1. A composite human state-of-the-art baseline averages 44.4.
What makes this disruptive
Retrieval-augmented generation is a settled idea. Building a good system is not. Effective choices vary by task and domain, and the work usually takes repeated engineering experiments. AutoRAGBench moves the question from whether a model can answer with retrieved text to whether an agent can construct the retrieval system itself. The authors present it as the first benchmark that asks for that construction from scratch in an open code space, rather than inside a fixed template. The from-scratch result and the improvement result answer different questions. From scratch, the leading configuration averages 56.1 against a composite human baseline of 44.4. Starting from a human-designed system on each of seven tasks, the same configuration improves the score by 16.6 points on average.
Why it matters (outside the lab)
Teams that ship retrieval systems spend much of their time on pipeline choices, not on the language model alone. If an agent can analyze task data, write the pipeline, and revise it from experimental feedback inside a fixed time and hardware budget, some of that iteration can be automated. The abstract's conclusion is that language-model agents can automate effective development of these systems. The scores also show the limit of that sentence. The leading average is 56.1, and the human composite baseline is 44.4, so the agent lead is real on the benchmark's own averages and is not described as a solved engineering problem. The separate 16.6 point gain matters because it starts from a human design on seven tasks, which is closer to how a team would actually hand work to an agent.
Limitations & open questions
The abstract does not define the score scale, so 56.1 and 44.4 should not be read as percentages or as fractions of a stated maximum. It does not name the nine tasks, the nine agent configurations, or the seven tasks used in the from-human comparison. It does not show the development trajectories in any detail, even though it says those trajectories were analyzed. It does not report cost beyond the six-hour budget on one H800 GPU, and it does not show how the systems behave outside this benchmark. The two numeric comparisons are different setups and are not one improvement figure.
Explain ladder
Default article depth
A retrieval-augmented system looks up external documents and then asks a language model to answer from them. Building a good one still takes trial and error, and the right design depends on the job. AutoRAGBench makes agents do that building with no starter code, for nine tasks, within six hours on one H800 GPU. The best reported configuration, GPT-6.1 Sol with Codex CLI, averages 56.1. A composite human baseline averages 44.4. On seven tasks, starting from an existing human design, that same configuration improves the score by 16.6 points on average.
Key terms
- Retrieval-augmented generation
- A setup that augments a large language model with information retrieved from an external collection of documents. The paper abbreviates it as RAG and treats the construction of these systems as the engineering problem under test.
- Open code space
- The agent writes the retrieval system as code, from scratch, rather than filling in a template. The benchmark also withholds starter code and prescribed strategies.
- Human-SOTA baseline
- The composite human state-of-the-art baseline in this benchmark. The abstract reports its average as 44.4 and does not describe how the composite was assembled.
- H800
- The graphics processor named as the single device each agent may use. The time budget on that device is six hours per task. The abstract does not give further hardware specifications.
Sources
Related explainers
Same topic and week first — keep exploring the scarcity → abundance map.
Imagine a new viewpoint only when the picture is not enough
2026-W48 · score 93 · Artificial Intelligencesame weeksame topic
A materials model trained to read text, structures, and images together
2026-W48 · score 91 · Artificial Intelligencesame weeksame topic
A training gym where chip-design agents keep state across tools
2026-W48 · score 89 · Artificial Intelligencesame weeksame topic
A fluent clinical answer can still rest on reasoning nobody checked
2026-W48 · score 87 · Artificial Intelligencesame weeksame topic
How Deep is the Signal? Time-Context up to 30 Years for Colorectal Cancer Prediction in a Primary-Care EHR Model
2026-W46 · score 93 · Artificial Intelligencesame topic
Disruptiveness
Editorial triage 0–100 · not peer review
- Novelty100
- Impact92
- Field heat97
- Practicality83
- Controversy45
