Skip to content

Conversation

@jc200808
Copy link
Contributor

OpenJudge Version

[The version of OpenJudge you are working on, e.g. import openjudge; print(openjudge.__version__)]

Description

[Please describe the background, purpose, changes made, and how to test this PR]

Checklist

Please check the following items before code is ready to be reviewed.

  • Code has been formatted with pre-commit run --all-files command
  • All tests are passing
  • Docstrings are in Google style
  • Related documentation has been updated (e.g. links, examples, etc.)
  • Code is ready for review

@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @jc200808, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refines the openjudge grading system by standardizing the use of default templates across various graders, ensuring consistent behavior when custom templates are not supplied. It also improves the readability and consistency of multimodal grader prompts through better formatting and updates their asynchronous usage examples for clarity and correctness.

Highlights

  • Default Template Initialization: Many graders across agent, common, and multimodal categories now automatically use a default template if no custom template is explicitly provided during initialization, enhancing robustness and reducing boilerplate.
  • Multimodal Prompt Formatting Improvement: Prompt strings for multimodal graders (ImageCoherence, ImageHelpfulness, TextToImage) are now consistently formatted using textwrap.dedent and .strip() to ensure clean, well-aligned prompts.
  • Updated Asynchronous Examples: Docstring examples for multimodal graders have been updated to correctly demonstrate asynchronous evaluation using asyncio.run(), making them directly runnable.
  • Typo Correction: A minor grammatical typo in the description of the tool_selection grader has been corrected.
  • Exported Default Templates: The default templates for ImageHelpfulnessGrader and TextToImageGrader are now explicitly included in their respective __all__ lists, making them accessible for external use.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a series of updates across multiple grader classes, enhancing robustness and consistency. Key changes include ensuring default templates are correctly applied in grader initializers, removing redundant code, and improving docstring examples to be runnable. The prompt definitions in multimodal graders have also been refactored for better readability.

Overall, the changes are positive and improve the codebase. I've pointed out a minor but important correction in two of the multimodal graders where an incorrect parameter name was used in the super().__init__ call. Addressing this will ensure the code is not just functionally correct but also clear and maintainable.

"""
super().__init__(
name="image_coherence",
grader_mode=GraderMode.POINTWISE,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

It seems there's a small typo in the parameter name. The LLMGrader's __init__ method expects the parameter mode, but grader_mode is used here. While this might work due to **kwargs handling, it's better to use the correct parameter name for clarity and to avoid potential issues.

Suggested change
grader_mode=GraderMode.POINTWISE,
mode=GraderMode.POINTWISE,

"""
super().__init__(
name="image_helpfulness",
grader_mode=GraderMode.POINTWISE,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Similar to another file in this PR, there's a small typo here. The LLMGrader's __init__ method expects the parameter mode, but grader_mode is used. Using the correct parameter name mode will make the code more robust and clear.

Suggested change
grader_mode=GraderMode.POINTWISE,
mode=GraderMode.POINTWISE,

Copy link
Collaborator

@ployts ployts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@helloml0326 helloml0326 merged commit 2ade760 into modelscope:main Jan 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants