Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add inverse chat templating #33321

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

Add inverse chat templating #33321

wants to merge 11 commits into from

Conversation

Rocketknight1
Copy link
Member

@Rocketknight1 Rocketknight1 commented Sep 5, 2024

Very experimental PR for now! This PR adds inverse chat templating, where we convert a formatted chat back into message dicts in a universal format. This has been requested several times, and it's critical to allow seamless tool use in pipelines, without requiring users to manually write parsers for each model.

Inverse templating requires the ability to extract text from a large input string. After testing several templates, I found that we can generally handle this by exposing two functions in the inverse templating environment: a slightly modified re.finditer() and a totally unmodified json.loads()

TODO:

  • Make some model PRs to test with this, but don't merge yet! (PRs not open, but templates written)
  • Add tests for loading/saving of inverse templates
  • Add tests for inverse template function
  • Test recovery of tools as well as messages
  • Make sure I don't need any extra functions
  • Refactor chat template tests out of tokenization_common so they're not run for every model
  • Add chat template tests to CircleCI
  • Make sure extraction works correctly with generation, and add some tests that use (static!) generation outputs
  • Add tool use to pipelines (will put this in a separate PR)

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@LysandreJik
Copy link
Member

Thanks @Rocketknight1! I'll review ASAP

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants