Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/eval spans #132

Draft
wants to merge 7 commits into
base: develop
Choose a base branch
from
Draft

Feature/eval spans #132

wants to merge 7 commits into from

Conversation

laurejt
Copy link
Contributor

@laurejt laurejt commented Jan 9, 2025

Here is the current code for evaluating span annotations.

The script requires two key files to run, page-level JSONL files that contain reference and system span annotations.

Usage: python evaluate_poetry_spans ref_jsonl sys_jsonl out.csv

Note that I'm considering adding reporting for the following:

  • number of exact matches
  • number of partial matches
  • number of missed reference spans
  • number of spurious system spans

@laurejt laurejt requested a review from rlskoeser January 9, 2025 19:06
Copy link

codecov bot commented Jan 9, 2025

Codecov Report

Attention: Patch coverage is 96.58247% with 23 lines in your changes missing coverage. Please review.

Project coverage is 80.81%. Comparing base (431b528) to head (a0bce13).

Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #132      +/-   ##
===========================================
+ Coverage    75.12%   80.81%   +5.69%     
===========================================
  Files           21       23       +2     
  Lines         1865     2538     +673     
===========================================
+ Hits          1401     2051     +650     
- Misses         464      487      +23     

@laurejt
Copy link
Contributor Author

laurejt commented Jan 10, 2025

Codecov Report

Attention: Patch coverage is 99.45652% with 2 lines in your changes missing coverage. Please review.

Project coverage is 99.64%. Comparing base (cb03ab8) to head (718aa4b).
Report is 2 commits behind head on develop.

Additional details and impacted files

Looks like Codecov is only checking the test file and not the actual code file.

@jerielizabeth jerielizabeth mentioned this pull request Jan 13, 2025
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: To Do
Development

Successfully merging this pull request may close these issues.

1 participant