Skip to content

Add DeepAnalyze system#26

Open
LIUyizheSDU wants to merge 3 commits intomitdbg:mainfrom
LIUyizheSDU:main
Open

Add DeepAnalyze system#26
LIUyizheSDU wants to merge 3 commits intomitdbg:mainfrom
LIUyizheSDU:main

Conversation

@LIUyizheSDU
Copy link

This pull request introduces the new DeepAnalyze System (KramaBench) and provides all necessary scripts, configuration, and documentation to integrate, evaluate, and run it within the project. The changes include adding environment setup and execution scripts, a detailed README, a prompt template for the system, and registering the system within the project.

Key additions and changes:

DeepAnalyze System Integration:

  • Registered the DeepAnalyze system in the project by updating systems/__init__.py to import it, making it available for evaluation workflows.

Execution and Evaluation Scripts:

  • Added run.sh for running DeepAnalyze across multiple or specific workloads, including logic to clean the data directory using a snapshot, run evaluations in parallel, and summarize scores.
  • Added eval_response_cache.sh to re-evaluate cached responses without regenerating outputs, facilitating fast recomputation of scores.

System Prompt:

  • Introduced prompt.py containing the DeepAnalyze prompt template, specifying instructions and answer formatting for the system's operation.

Documentation:

  • Added a README.md for DeepAnalyze, explaining environment setup, running workloads, output locations, and cache evaluation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant