Skip to content

Conversation

@svempati21
Copy link

@svempati21 svempati21 commented Oct 19, 2023

@cthoyt
Copy link
Collaborator

cthoyt commented Oct 24, 2023

Next steps:

  1. Create a 4 column spreadsheet (e.g., with annotations from SciWheel) with example sentences, subject, predicate, and object that will serve as a "benchmark" corpus. This probably needs 20-30 examples to start
  2. We're going to run the workflow on these sentences and make a comparison of how often they get the right answer (based on the human curated benchmark). We can then calculate the accuracy (and other metrics) based on that.
  3. (optional) If we want to use metrics that incorporate false positives, we can also include 20-30 random strings from the same corpus but that don't have any relations that should get extracted

@bgyori
Copy link
Member

bgyori commented Nov 15, 2023

I just noticed - this work doesn't really belong in this repository. It should be in the https://github.com/gyorilab/indra_spine repository since that's where this line of work has been developed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants