Skip to content

Issues: EvolvingLMMs-Lab/lmms-eval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

187 Open 80 Closed
187 Open 80 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Get N/A when evaluating lmms-lab/llava-onevision-qwen2-7b-ov on mathvista_testmini, docvqa_test, and infovqa_test
#553 opened Feb 24, 2025 by viyjy
Bug in qwen2-vl ?
#552 opened Feb 23, 2025 by LiJunscs
Support of VideoChat-Flash
#551 opened Feb 22, 2025 by hello-bluedog
new
#549 opened Feb 21, 2025 by Marcher-lam
How to use LLM as judge?
#548 opened Feb 21, 2025 by whc688
[Request] Support for MileBench
#547 opened Feb 21, 2025 by commmet-ahn
gen_kwargs['until'] defaults to "\n\n"
#543 opened Feb 19, 2025 by nanocm
Use --use_cache but the responses are not saved
#542 opened Feb 17, 2025 by XXXCARREY
Any plan for supporting ""LVBench: An Extreme Long Video Understanding Benchmark""
#539 opened Feb 14, 2025 by RexGPT
MMMU-test doesn't generate submission files
#538 opened Feb 13, 2025 by zjuAJW
AttributeError: 'dict' object has no attribute 'to_dict'
#537 opened Feb 12, 2025 by xiGHK
MLVU aggregate function is incorrect
#534 opened Feb 11, 2025 by SCZwangxiao
1
MLVU aggregate function is incorrect
#533 opened Feb 11, 2025 by SCZwangxiao
video mme
#532 opened Feb 11, 2025 by Violettttee
1
TypeError: 'JpegImageFile' object is not iterable
#512 opened Jan 27, 2025 by lmx760581375
2
Qwen2-VL,IndexError: index 0 is out of bounds for dimension 0 with size 0
#506 opened Jan 19, 2025 by cxli2333
long time waiting after postprocessing
#504 opened Jan 17, 2025 by yiyexy
1
reproduce the LLaVA-NeXT-Video 7B scores
#503 opened Jan 16, 2025 by junho-jo
1
[OOM] CUDA out of memory error during evaluating llava-hf and llama-vision models
#500 opened Jan 14, 2025 by commmet-ahn
2
ValueError: BuilderConfig 'testdev_balanced_instructions' not found. Available: ['default']
#499 opened Jan 14, 2025 by lmx760581375
2
load_video gets 0 frame for mp4 file
#495 opened Jan 11, 2025 by yuanyehome
@pufanyi
[Type error] cache_requests parameter is mismatch.
#493 opened Jan 9, 2025 by minimi-kei
1
Evaluation is getting slower
#491 opened Jan 7, 2025 by Yibo-0820
2
How can I evaluate specific MME subtasks
#490 opened Jan 6, 2025 by whyisverysmart
2
eval on multi-gpus
#482 opened Dec 29, 2024 by masn1310
1
ProTip! What’s not been updated in a month: updated:<2025-01-24.