-
Notifications
You must be signed in to change notification settings - Fork 45
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add a proposal of Smart Coding benchmark suite #120
base: main
Are you sure you want to change the base?
Conversation
Signed-off-by: boX <[email protected]>
The PR should further elaborate on how to implement this feature in Ianvs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This proposal is related to #98 and there are two advices.
- Putting ianvs integration at the final stage would be risky for this project. The integration shall be started asap. As mentioned in the routine meeting, a recommended implementation from @IcyFeather233 is in https://github.com/IcyFeather233/ianvs/tree/dev, one can follow this readme to try llm single task learning and open compass.
- It is not clear how would it be related to the SIG AI project. An architecture with ianvs is needed. E.g., considering dataset, metric in TestEnv Manager. You might want to take a look at https://github.com/kubeedge/ianvs/pull/122/files for architecture example.
Signed-off-by: boX <[email protected]>
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Improve the architecture diagram Signed-off-by: boX <[email protected]>
# Conflicts: # docs/proposals/test-reports/Smart Coding benchmark suite Proposal.md
Signed-off-by: boX <[email protected]>
The logic of the subject is basically consistent with Meng Zhuo's government affairs model. |
Smart Coding benchmark suite: built on KubeEdge-lanvs
What type of PR is this?
/kind test
/kind design
What this PR does / why we need it:
The PR is a proposal to enhance a dataset by annotating the core code of the classic Python project, and then add project problems and App projects, etc., to finally complete the creation of the dataset and conduct test evaluation.
Which issue(s) this PR fixes:
#98
Fixes #