Unbundling Multimodal Vision Encoding

The 30-second take

• A vision encoder trained jointly from scratch matches a pretrained encoder when data and lifetime compute are held fixed.
• Removing the encoder still hurts visual questions, even if the language backbone is enlarged by the encoder's parameter count.
• People training multimodal models can skip a separate vision-pretraining stage and still keep a dedicated visual module.

What the paper actually did

The paper separates three choices that usual vision encoders bundle together: a visual prior from earlier pretraining, dedicated vision capacity, and placing that capacity before the multimodal backbone. Published models also differ in data, architecture, and compute, so those comparisons cannot isolate each choice. Here every model is pretrained from scratch on common data and compared on visual question answering and language benchmarks at matched lifetime compute. The visual prior is not necessary: an encoder trained jointly from scratch is on par with a pretrained one, without a separate training stage. Dedicated vision capacity does matter. An encoder-free model trails on visual understanding even when its backbone is enlarged by the encoder's parameter count. That capacity need not sit before fusion. Vision-only modules added inside early backbone layers perform comparably to a pre-fusion encoder and can lower time to first token. The pattern holds across backbone sizes, when encoder pretraining is treated as free, and when the run starts from a pretrained language model.

What makes this disruptive

The expensive habit is the separate vision model that must be trained, stored, and then fused. This comparison says the habit is not what buys visual skill once compute and data are matched. Keeping a block of parameters that only see vision, and training it in the same run as the backbone, is enough. Putting those parameters inside early layers, rather than in front of the model, can also shorten time to first token.

Why it matters (outside the lab)

Teams building multimodal systems often inherit a pretrained vision tower because that is how the last strong model was built. If a jointly trained block matches that tower, they can spend the same compute on one training run instead of a vision stage plus a language stage. The result is a simpler recipe, not a claim that every existing encoder should be thrown out of production tomorrow.

Limitations & open questions

The abstract reports the comparisons as on par, trailing, or comparable. It does not list dataset names, model sizes, or the numeric gap on each benchmark. Matched lifetime compute is inside this controlled framework. Treating encoder pretraining as free is a separate check, not a statement that pretraining was actually free. Time to first token can be lower for the in-fusion placement; the abstract does not give the latency.