Skip to content

QA dataset generation and rise experiment #175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

QA dataset generation and rise experiment #175

wants to merge 13 commits into from

Conversation

@shackmann
Copy link
Contributor

@shackmann shackmann commented Jul 23, 2025 •
edited
Loading

  • notebook to generate a new HF QA dataset from raw text (potentially private)
  • new private dataset in storage
  • experiment based on the new dataset

@shackmann shackmann linked an issue Jul 23, 2025 that may be closed by this pull request
QA dataset generation #174
Open
@shackmann shackmann requested review from alex-dr and mhauskn-dr and removed request for alex-dr and mhauskn-dr July 23, 2025 17:50
@shackmann shackmann requested review from alex-dr and mhauskn-dr July 24, 2025 13:19
@shackmann shackmann self-assigned this Jul 24, 2025
alex-dr
alex-dr reviewed Jul 28, 2025
View reviewed changes
syftr/storage.py Outdated
supporting_facts=[],
difficulty="default",
qtype="default",
gold_evidence=[],
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dataset generation script seems to populate this field - we should include it here.

alex-dr
alex-dr reviewed Jul 28, 2025
View reviewed changes
Copy link
Collaborator

@alex-dr alex-dr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mike had some logic for reviewing and filtering generated QA pairs after generation. From what I've seen, we can end up with partially generated answers and stuff, so it'd be good if we can look up what he was doing and incorporate it into the notebook.

@shackmann shackmann requested a review from alex-dr August 4, 2025 15:01
alex-dr
alex-dr reviewed Sep 11, 2025
View reviewed changes
Copy link
Collaborator

@alex-dr alex-dr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks okay, but we need to rebase this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alex-dr alex-dr alex-dr left review comments

@mhauskn-dr mhauskn-dr Awaiting requested review from mhauskn-dr

At least 1 approving review is required to merge this pull request.

Assignees

@shackmann shackmann

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

QA dataset generation

3 participants

@shackmann @alex-dr @debadeepta