Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Text to Image metric incorrect evaluation #1251

Open
shanmukh-sai2003 opened this issue Jan 7, 2025 · 3 comments
Open

Text to Image metric incorrect evaluation #1251

shanmukh-sai2003 opened this issue Jan 7, 2025 · 3 comments

Comments

@shanmukh-sai2003
Copy link

❗BEFORE YOU BEGIN❗
Are you on discord? 🤗 We'd love to have you asking questions on discord instead: https://discord.com/invite/a3K9c8GRGt

Describe the bug
In Text to image metric while evaluating Semantic Consistency we should pass the input text prompt and actual output image to the model for evaluation. Instead, we are passing the input text prompt and image from the input if it contains any.

Expected behavior
We should pass the actual output image and input text prompt to the model for evaluation.

Screenshots
image

@kritinv
Copy link
Collaborator

kritinv commented Jan 8, 2025

Hey @shanmukh-sai2003, thanks for pointing it out. It should be fixed here: #1253

@shanmukh-sai2003
Copy link
Author

thanks for the fix @kritinv. Can I know when I can expect this fix to be released

@penguine-ip
Copy link
Contributor

@shanmukh-sai2003 I believe its already out, can you give it a try? :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants