Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Make language a list in evaluate app, added comments to evaluate description #25

Merged
merged 2 commits into from
Feb 2, 2024
Merged

fix: Make language a list in evaluate app, added comments to evaluate description #25

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c8f33ba
Reworked code so that it can run the metric. Added comments for fello…
fasterinnerlooper Jan 31, 2024
4e0c18a
Merge branch 'main' into FixLanguageArrayIssue
k4black Feb 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions evaluate_app/codebleu.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@
should be a string with tokens separated by spaces.
references: list of reference for each prediction. Each
reference should be a string with tokens separated by spaces.
language: programming language in ['java','js','c_sharp','php','c','python','cpp'].
language: programming language in ['java','js','c_sharp','php','c','python','cpp']. Please note that, due to the way Datasets works, the number of entities in the language array must match the number of entries in the predictions and references arrays, but only the first value from the languages array will be used. This means that you will not be able to compute a metric for different langauges at the same time, but mst do them as sequential calls to CodeBleu.
weights: tuple of 4 floats to use as weights for scores. Defaults to (0.25, 0.25, 0.25, 0.25).
Returns:
codebleu: resulting `CodeBLEU` score,
Expand All @@ -53,7 +53,7 @@
>>> metric = evaluate.load("k4black/codebleu")
>>> ref = "def sum ( first , second ) :\n return second + first"
>>> pred = "def add ( a , b ) :\n return a + b"
>>> results = metric.compute(references=[ref], predictions=[pred], language="python")
>>> results = metric.compute(references=[ref], predictions=[pred], language=["python"])
>>> print(results)
"""

Expand All @@ -76,7 +76,7 @@ def _info(self):
{
"predictions": datasets.Value("string", id="sequence"),
"references": datasets.Sequence(datasets.Value("string", id="sequence"), id="references"),
"lang": datasets.Value("string"),
# "lang": datasets.Value("string"),
# "weights": datasets.Value("string"),
# "tokenizer": datasets.Value("string"),
}
Expand All @@ -85,7 +85,7 @@ def _info(self):
{
"predictions": datasets.Value("string", id="sequence"),
"references": datasets.Value("string", id="sequence"),
"lang": datasets.Value("string"),
# "lang": datasets.Value("string"),
# "weights": datasets.Value("string"),
# "tokenizer": datasets.Value("string"),
}
Expand Down Expand Up @@ -113,7 +113,7 @@ def _compute(self, predictions, references, lang, weights=(0.25, 0.25, 0.25, 0.2
return self.codebleu_package.calc_codebleu(
references=references,
predictions=predictions,
lang=lang,
lang=lang[0],
weights=weights,
tokenizer=tokenizer,
)
Loading