Existing benchmarks primarily report aggregate accuracy, which obscures whether errors stem from perception, reasoning, or their interaction. Some vary input modalities, but without strict controls they fail to isolate components.
Fine-grained probes test perception, and the textual description tests reasoning under trivialized perception. Errors that remain on the multimodal question when both succeed are multimodal-specific.
There are triangle BDF, triangle FIA, line BG, line CI. GBFI are collinear. AFD are collinear. AIC are collinear. ∠BDF=47°. ∠CIF=112°. ∠DFB=65°.
Find measure of angle IAF.
The Perception Test evaluates questions answerable directly from the diagram, such as reading an annotated angle.
The Reasoning Test replaces the diagram with a complete textual description, such that the question can be solved without visual access.
Multimodal-specific: correct perception and text in isolation but failure on the combined multimodal task. Figures label it “Integration” or “Interaction”.
Reasoning: correct perception probes but failed text-based reasoning.
Perception: failure on perception probes but correct reasoning from text.
Perception & Reasoning: failures on both perception probes and textual reasoning.
Correct: counted as Trivial (solvable from text alone) or Rest (all other correct cases).
Starting from the symbolic semantic state, we build four aligned annotations: diagrams rendered from geometric constraints, a definitionally equivalent textual description, multimodal questions requiring both modalities, and fine-grained probes targeting recovery of visual details.
The question drops clauses that overlap with the diagram, so it requires the visual context. New diagrams are rendered from the semantic state to avoid visual familiarity effects.
To test robustness against visual modifications, semantic diagram modifications alter visual form while preserving task correctness.
Finding 1
With strong textual SFT, multimodal RL mainly boosts perception; without it, RL modestly improves both modalities.
Performance gains by input modality. Bars show percentage point shifts from finetuning for text versus diagram inputs.
Finding 2
Textual SFT does not use visual inputs, yet it enhances perceptual performance. Reflective reasoning is more frequent in predictions that became accurate after training than in those that were already accurate.
Perception performance shifts from finetuning in percentage points. Except for multimodal SFT, all methods improve perception.
Sample distribution over reflection count. Reflection is more frequent in false→true cases than in true→true cases, showing that perception gains from textual SFT arise partially from reflective reasoning.
Finding 3
RL primarily reduces perception and reasoning errors on the same problems, but many of the reductions manifest as multimodal-specific residuals.
Distribution of error types per model. Most gains come from cases failing in both perception and reasoning. Multimodal-specific residual errors become more prominent as the other categories improve.
Finding 4
Multimodal RL improves consistency, the expected agreement of predictions across perturbations of the same diagram, whereas multimodal SFT reduces it.
Robustness to semantic-level visual modifications. Box plots show accuracy on modified diagrams. Points report accuracy on the unmodified base diagrams and the overall consistency score.
While most training methods are stable under familiar diagrams, multimodal SFT suffers a sharp drop on newly rendered diagrams.
Finetuning shifts by diagram type. Bars show percentage point changes for existing versus OOD rendered diagrams.
Finding 5
Multimodal reasoning training preferentially improves skills driven by direct geometric cues, while perception tasks that rely on symbolic marks or long-range visual associations show limited gains.
Perception probe results by question type.
Impact of multimodal reasoning training on MathLens. We evaluate pretrained backbones alongside models finetuned for multimodal reasoning, reporting accuracy (%).
The backbone model fails in perception, leading to an incorrect answer. The textual SFT model initially makes the same mistake but recovers the perceptual detail and produces the correct answer.
The backbone model exhibits a reasoning error, whereas the multimodal RL model provides the correct solution.
The textual SFT model corrects the perceptual error present in the backbone model.
The multimodal RL model inherits the backbone model's flawed reasoning, leading to perception errors and downstream failure.
The multimodal RL model corrects the perceptual error of the backbone model but ultimately fails to produce the correct answer due to a reasoning error.
We curate problem instance from a prior dataset (Fu et al., 2024) and annotate the text description and perception-centric question-answers.
@inproceedings{chung2025whatmllms,
title={What MLLMs Learn about When they Learn about Multimodal Reasoning},
author={Chung, Jiwan and Joshi, Neel and Sharma, Pratyusha and Yu, Youngjae and Vineet, Vibhav},
booktitle={Advances in Neural Information Processing Systems: NeurIPS},
year={2026}
}