-
Notifications
You must be signed in to change notification settings - Fork 15
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Make it easier to run search with custom models (#97)
Our single-step evaluation CLI defines the evaluation-specific arguments in `BaseEvalConfig`, and then creates `EvalConfig` by combining `BaseEvalConfig` and `BackwardModelConfig` as base classes. This design allows one to inject custom models by defining a modified `BackwardModelConfig`, then creating a modified `EvalConfig` (by mixing in `BaseEvalConfig`), and passing that to the syntheseus code (relying on a bit of duck typing). Internally, this allowed us to easily benchmark internal models using the shared eval script from syntheseus. This PR adapts the search CLI to follow a similar convention, so that one can also easily integrate custom models into search.
- Loading branch information
Showing
2 changed files
with
16 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters