Towards Versatile Visual Document Grounding and Referring
Yinan Zhou*, Yuxin Chen*, Haokun Lin, Yichen Wu, Shuyu Yang, Li Zhu‡, Zhongang Qi‡, Chen Ma‡, Ying Shan
*Equal Contribution †Project Lead ‡Corresponding Authors
📖Arxiv 📽️Demo Video 🐶Project Page
In recent years, Multimodal Large Language Models (MLLMs) have increasingly emphasized grounding and referring capabilities to achieve detailed understanding and flexible user interaction. However, in the realm of visual document understanding, these capabilities lag behind due to the scarcity of fine-grained datasets and comprehensive benchmarks. To fill this gap, we propose the DOcument Grounding and Referring data engine (DOGR-Engine), which produces two types of high-quality fine-grained document data: multi-granular parsing data for enhancing fundamental text localization and recognition capabilities; and instruction-tuning data to activate MLLM's grounding and referring capabilities during dialogue and reasoning. Additionally, using our engine, we construct DOGR-Bench, which encompasses 7 grounding and referring tasks across 3 document types (chart, poster, PDF document), providing comprehensive evaluations for fine-grained document understanding. Furthermore, leveraging the data generated by our engine, we develop a strong baseline model, DOGR. This pioneering MLLM is capable of accurately referring and grounding texts at multiple granularities within document images.
2025-08-06: training and inference code (this repository) are released.
2025-08-06: Accepted to ICCV 2025.
Watch the introduction video here!
Our code is based on LLaVA-NeXT. Installation can be found in LLaVA-NeXT
bash scripts/eval/dogr_bench_inference_ddp.shscripts/eval/dogr_evaluation.shThe model weights are in MODEL WEIGHTS
python inference/demo_gradio.py@misc{zhou2025dogrversatilevisualdocument,
title={DOGR: Towards Versatile Visual Document Grounding and Referring},
author={Yinan Zhou and Yuxin Chen and Haokun Lin and Yichen Wu and Shuyu Yang and Zhongang Qi and Chen Ma and Li Zhu and Ying Shan},
year={2025},
eprint={2411.17125},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2411.17125},
}
Please refer to our LICENSE for more license details.