网站介绍
全方位的多模态大模型能力评测体系
官网公开介绍
MMBench online evaluation service will cease operations starting March 31, 2026, and the corresponding leaderboard will no longer be updated.×
Based on perception and reasoning
progressively refining evaluation dimensions
Approximately 3000 multiple-choice questions covering object detection, text recognition, action
recognition, image caption, relation reasoning, and 20 other fine-grained evaluation dimensions
A More Robust Evaluation Approach
Repeating the same multiple-choice question with shuffled options, models providing consistent answers are considered
passed. Compared to traditional 1-pass top-1 accuracy evaluation, the average accuracy decreases by 10% ~ 20%.
Minimizing the impact of noise on evaluation results to ensure the reproducibility.
The original VL problem:
Circular Evaluation
Q: How many apples are there in the image?
A.4B.3C.2D.1
4 Passes in Circular Evaluation (choices with circular shift):
1. Q: How many apples are there in the image? 2. Q: How many apples are there in the image? 3. Q: How many apples are there in the image? 4. Q: How many apples are there in the image?
Choices: A.4 B.3 C.2 D.1Choices: A.3 B.2 C.1 D.4Choices: A.2 B.1 C.4 D.3Choices: A.1 B.4 C.3 D.2
VLM prediction: DVLM prediction: CVLM prediction: AVLM prediction: A
GT: DGT: CGT: BGT: A
VLM failed at pass 3. Thus wrong.
A More Reliable Model Output Extraction Method
Matching model outputs with options based on ChatGPT, even if the model does not output
according to the instructions, it can accurately match to the most reasonable option.
Meta Instruction
You are an Al assistant to help me matching an answer with several options of a single-choice question. \nYou are provided with a question, several options, and an answer, \nand you need to find which option is most similar to the answer. \nIf the meaning of all options are significantly different from the answer, output E. \nYou should output a single uppercase character in A, B, B, D (if they are valid options), and E. \n