EpiSEM-Bench: Evaluating Safety and Evidence Recovery in LLM Agents under Missing Evidence

The 30-second take

• What: EpiSEM-Bench is an executable test of 300 matched unsafe and safe task pairs across six domains that measures whether tool-using language-model agents act unsafely when a critical fact is omitted, and whether they recover that evidence.
• Why it matters: Safe tool use is still a scarce, hand-audited luxury. If omission robustness and evidence recovery become default test scores, agents that look fluent on complete context may stop being treated as ready to act.
• Who should care: Teams shipping tool-using agents, safety evaluators, and anyone who currently trusts an agent because every tool reply it saw was true.

What the paper actually did

The authors study tool-using language-model agents that rely on environmental facts to decide whether an action is permitted. When a tool response omits a safety-relevant fact, they note, an agent may carry out a prohibited action even though the user request is benign and every returned statement is true.

They introduce EpiSEM-Bench, named from the title (the abstract's typesetting macro is a placeholder for the benchmark), an executable suite for evaluating unsafe execution, safety boundaries, and evidence recovery in large language model agents under missing evidence. It contains 300 matched unsafe and safe task pairs across six domains. By systematically suppressing critical facts, the benchmark measures unsafe execution under critical evidence omission and determines the minimum observed failure cardinality, including failures with full evidence and cases with no observed failure after exhaustive testing.

An independent audit tool tests whether agents seek missing facts and then withhold prohibited actions while completing permissible tasks. On ten models, they report substantial variation in omission robustness, evidence acquisition, and appropriate action. Resistance to missing evidence does not necessarily translate into useful recovery. Recovery access reduces unsafe execution for all ten models, but agents differ in whether they actively obtain the missing facts and use them to act appropriately in both worlds.

What makes this disruptive

If the core claim holds, the statement that the tool told the truth is no longer a safety story. The scarce capability under pressure is an agent that notices a missing permit-or-prohibit fact, goes looking for it, and then withholds the forbidden action while still finishing the allowed one.

The distinctive move is a matched unsafe-and-safe design plus systematic suppression of critical facts, so you can measure omission failures, the smallest set of missing evidence that still causes failure, and whether recovery access actually changes behavior. Variation across ten models, and the split between omission resistance and useful recovery, is the practical signal.

This week's editorial ranking placed the preprint third of twenty, with a score of 87. That is triage for the Disruptive Concepts brief, not a peer-review verdict. This is a diagnostic benchmark, not a new safety layer you can install.

Why it matters (outside the lab)

Abundance lens (today's luxuries toward tomorrow's defaults): trustworthy tool-using agents are still an elite, expensive audit product. If missing-evidence tests become ordinary, safer action under incomplete tools can move from a specialist red team toward a cheaper default score on the path to capable assistance as software rather than scarce human review.

Near-term, the preprint is a test harness for labs comparing ten models on omission, acquisition, and appropriate action, not a claim that any named model is deployment-safe. Medium-term, independent replication, domain coverage, and whether recovery stays cheap decide whether this protocol becomes a default gate.

No calendar date is implied. Disruptive Concepts, which publishes these explainers free for humans, reads the work as a possible brick toward making scarce safety review go further, not as a cleared product.

Limitations & open questions

This is a preprint benchmark, not a safety certification. Results cover 300 matched pairs across six domains and ten models as stated in the abstract; we have not rerun the suite. The abstract does not name the ten models, the six domains, or the size of the unsafe-execution reductions when recovery access is granted.

Minimum observed failure cardinality is a protocol outcome, including cases with full evidence and cases with no observed failure after exhaustive testing. No observed failure is not a proof of safety. The independent audit tool is described functionally, not as a released product with a version number here.

Abundance is not automatic. A harder test does not demonetize human oversight on a fixed date. Cost, coverage, and independent audits still sit between this benchmark and a default.