Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add a proposal of Smart Coding benchmark suite #120

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

safe-b
Copy link

@safe-b safe-b commented Jul 17, 2024

Smart Coding benchmark suite: built on KubeEdge-lanvs

What type of PR is this?
/kind test
/kind design

What this PR does / why we need it:
The PR is a proposal to enhance a dataset by annotating the core code of the classic Python project, and then add project problems and App projects, etc., to finally complete the creation of the dataset and conduct test evaluation.
Which issue(s) this PR fixes:
#98
Fixes #

@hsj576
Copy link
Member

hsj576 commented Jul 18, 2024

The PR should further elaborate on how to implement this feature in Ianvs.

Copy link
Collaborator

@MooreZheng MooreZheng Jul 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This proposal is related to #98 and there are two advices.

  1. Putting ianvs integration at the final stage would be risky for this project. The integration shall be started asap. As mentioned in the routine meeting, a recommended implementation from @IcyFeather233 is in https://github.com/IcyFeather233/ianvs/tree/dev, one can follow this readme to try llm single task learning and open compass.
  2. It is not clear how would it be related to the SIG AI project. An architecture with ianvs is needed. E.g., considering dataset, metric in TestEnv Manager. You might want to take a look at https://github.com/kubeedge/ianvs/pull/122/files for architecture example.

@kubeedge-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign moorezheng after the PR has been reviewed.
You can assign the PR to them by writing /assign @moorezheng in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubeedge-bot kubeedge-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Aug 14, 2024
@MooreZheng MooreZheng added kind/design Categorizes issue or PR as related to design. and removed proposal PR labels Aug 29, 2024
safe-b and others added 4 commits September 11, 2024 21:45
Improve the architecture diagram

Signed-off-by: boX <[email protected]>
# Conflicts:
#	docs/proposals/test-reports/Smart Coding benchmark suite Proposal.md
@safe-b
Copy link
Author

safe-b commented Sep 11, 2024

The logic of the subject is basically consistent with Meng Zhuo's government affairs model.
The comment data set of the code model is basically the same as that of the government model. The difference lies in the content of the issue data set. One question may have multiple answers.
The next step is training and testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/design Categorizes issue or PR as related to design. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants