Qusetion about math_vision and mmmu_pro evaluation result
May I ask: for MathVision, is the evaluation result reported as overall accuracy? And for MMMU-Pro, is the evaluation based on 10-choice (10 options) results?
Hi @JjjjjZzz Apologies for late response
Yes. MathVision (MATH-V) reports results as overall accuracy NeurIPS, aggregated across its 3,040 problems spanning 16 mathematical disciplines and 5 difficulty levels .
Reference : https://github.com/mathllm/MATH-V
MMMU-Pro rigorously assesses multimodal models’ true understanding and reasoning capabilities through a three-step process based on MMMU (1) filtering out questions
answerable by text-only models, (2) augmenting candidate options, and (3) introducing a vision-only input setting where questions are embedded within images. For more info pls refer this https://arxiv.org/pdf/2409.02813
Thanks