-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/eval spans #132
base: develop
Are you sure you want to change the base?
Feature/eval spans #132
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #132 +/- ##
===========================================
+ Coverage 75.12% 80.81% +5.69%
===========================================
Files 21 23 +2
Lines 1865 2538 +673
===========================================
+ Hits 1401 2051 +650
- Misses 464 487 +23 |
Looks like Codecov is only checking the test file and not the actual code file. |
Now also report counts of matches, misses, and spurious.
Here is the current code for evaluating span annotations.
The script requires two key files to run, page-level JSONL files that contain reference and system span annotations.
Usage:
python evaluate_poetry_spans ref_jsonl sys_jsonl out.csv
Note that I'm considering adding reporting for the following: