Skip to content

Releases: bigcode-project/bigcodebench

Release BigCodeBench v0.1.7

27 Jun 23:52
Compare
Choose a tag to compare

Fix some identified issues:

  • The ground truth pass rate was not previously computed in the correct way.
  • Passed RAM limits would raise errors, as they were set as float type.
  • User permission is not correctly set up in the Evaluate Docker.

Features:
-- check-gt-only will print out the pass rate when finishing.

Release BigCodeBench v0.1.6

26 Jun 21:39
Compare
Choose a tag to compare

New features;

  • The RAM setup is now adjustable via specific arguments.
  • Parallel ground truth checking is supported. Potentially failed checks are skipped during execution. A warning will be issued if the ground truth pass rate falls below 0.95.

Release BigCodeBench v0.1.5

18 Jun 13:31
Compare
Choose a tag to compare

New features;

  • The data is downloaded from HF hub by default.
  • Data formats have been unified for the one on HF and the one on GitHub.

BigCodeBench v0.1.2

08 Jun 13:16
Compare
Choose a tag to compare
fix: add known issues