feat: add Rust parser #31
Conversation
There was a problem hiding this comment.
Hi! Thank you so much for your great efforts!
Could you please fix couple of issues I mention in the review comment?
Additionally, could you please tests for rust in tests/test_codebleu.py file test_simple_cases_work_for_all_langs test case to be sure it is working on all devices.
| reference = "fn sum ( first , second )->i8 {\n second + first}" | ||
| result = calc_codebleu([reference], [prediction], lang="rust", weights=(0.25, 0.25, 0.25, 0.25), tokenizer=None) | ||
| print(result) | ||
|
|
There was a problem hiding this comment.
Could you please delete this file?
codebleu/codebleu.py
Outdated
| @@ -7,7 +7,7 @@ | |||
| from . import bleu, dataflow_match, syntax_match, weighted_ngram_match | |||
|
|
|||
| PACKAGE_DIR = Path(__file__).parent | |||
| AVAILABLE_LANGS = ["java", "javascript", "c_sharp", "php", "c", "cpp", "python", "go", "ruby"] # keywords available | |||
| AVAILABLE_LANGS = ["java", "javascript", "c_sharp", "php", "c", "cpp", "python", "go", "ruby", "rust"] # keywords available | |||
There was a problem hiding this comment.
Please also add "rust" to
- README.md
- evaluate_app/README.md
- evaluate_app/codebleu.py - description
| @@ -1213,3 +1213,176 @@ def DFG_javascript(root_node, index_to_code, states): | |||
| DFG += temp | |||
|
|
|||
| return sorted(DFG, key=lambda x: x[1]), states | |||
|
|
|||
|
|
|||
| def DFG_rust(root_node, index_to_code, states): | |||
There was a problem hiding this comment.
Please check pipeline failed, you need to apply black formatter (python -m black .)
Also pipeline if checking isort/ruff/mypy
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #31 +/- ##
==========================================
- Coverage 43.56% 41.27% -2.29%
==========================================
Files 11 11
Lines 1538 1696 +158
==========================================
+ Hits 670 700 +30
- Misses 868 996 +128 ☔ View full report in Codecov by Sentry. |
|
Could you take a look at the format fixes? Thanks -- Yijun |
|
@yijunyu Thanks for fixes! Could you please fix other issues i mentioned? Add README info about rust, remove extra file, add test case? |
|
Please do so. I am a bit tied up recently. The Readme won't be significantly different to those for existing languages. If you need help I can start with a similar document. I have a test case which I may add to the report for this, which fine shall I put it ? Or maybe I just paste it here. |
Adapted
DFG_csharptoDFG_rustin order to parse Rust code using thetree-sitter-rustparser.