Caption-once, Frames-on-Demand: Visual-Need Routing for Budget-Aware Agentic Long Video Understanding

The 30-second take

• What: Caption-once, Frames-on-Demand (CFD) builds an event-level story skeleton plus clip-level micro-log in one offline caption pass, then a cloud MLLM uses a Visual-Need Router to fetch bounded keyframes only for appearance, on-screen text, or attribute questions.
• Why it matters: Abundance angle: hours-long video understanding is still an elite compute-and-bandwidth luxury. Turning visual access into a query-conditioned cost is a step toward default long-video Q&A on tight budgets — near-term for systems, not a free consumer app date.
• Who should care: Edge-video and MLLM-agent teams, product owners of camera archives, and anyone paying for dense visual tokens on questions that were really about timeline structure.

What the paper actually did

Long-video understanding on edge devices must reason over hours of content under tight compute and bandwidth. Subsampling visual tokens loses temporal structure; text-only memories lose fine-grained visual attributes. The authors call this a visual-textual duality: language memories carry long-range temporal structure better than dense frames, while pixels remain decisive for attribute-level perception.

Caption-once, Frames-on-Demand (CFD) is a budget-aware edge–cloud agentic framework. The edge runs a single offline captioning pass that builds a dual-track narrative index — an event-level story skeleton plus a clip-level micro-log — cached and reused across queries without re-captioning. At query time, a cloud-side MLLM reasons over the index in a story-first loop centered on a Visual-Need Router: a per-query gate that triggers bounded keyframe retrieval only for perceptual questions (appearance, on-screen text, attribute disambiguation) and keeps temporal-structural questions in language space.

The router makes visual access a first-class, query-conditioned cost and caps per-query frame consumption regardless of video length. Experiments on long-video benchmarks, the abstract says, show strong accuracy–efficiency trade-offs while substantially reducing online visual processing.

What makes this disruptive

The architectural claim is that you should not pay visual tokens in proportion to video length. Caption once, index twice (events + micro-log), and spend frames only when the question is perceptual. If the router works, temporal questions stay cheap in language space and appearance questions pay a bounded keyframe tax.

The scarcity it touches is expert-grade attention over long video — today a compute and bandwidth luxury. A reusable edge index plus a cloud gate is a path toward default long-video assistance on constrained devices. That is a systems roadmap signal, not a claim that every camera feed becomes a free oracle.

The abstract reports “strong” trade-offs and reduced online visual processing without listing specific benchmark scores; stay with that level of claim until the PDF is read.

Why it matters (outside the lab)

Abundance lens (today’s luxuries → tomorrow’s defaults): Disruptive Concepts reads AI systems work as a move on a scarcity map — not as a finished product.

Scarcity today: expert judgment and analysis over long video that only expensive staff or cloud budgets can deliver.

If this line of work scales: capable video assistance as a default software layer rather than a scarce human or GPU service. Horizon: near-term (years, not decades) if reliability and cost keep improving — for agent architectures, not a promised consumer app.

Near-term: treat visual tokens as a gated cost; reuse a captioned index. Medium-term: caption quality, router mistakes, and privacy of the edge index decide whether this becomes a default. No invented launch year.

Limitations & open questions

This is a preprint. The abstract does not name the long-video benchmarks or quote accuracy numbers — only “strong accuracy-efficiency trade-offs” and reduced online visual processing. Router errors (fetching frames when unnecessary, or skipping them when appearance matters) are the obvious failure mode and must be checked in the PDF.

Offline captioning can drop or distort events; a cached index is only as good as that single pass. Edge–cloud split implies bandwidth and privacy design questions the abstract does not settle. Capping frames per query does not cap the one-time captioning cost of very long video.

Not yet a default: this does not demonetize cognitive video labor on a fixed date. Cost, reliability, and scale still sit between a framework paper and tomorrow’s default archive search.