Skip to content

Commit

Permalink
feat: add tokenizer-config-path to launcher args (#1495)
Browse files Browse the repository at this point in the history
This PR adds the `tokenizer-config-path` to the launcher and passes it
to the router

Fixes:
#1427
  • Loading branch information
drbh authored Jan 26, 2024
1 parent 650fea1 commit d975885
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 0 deletions.
8 changes: 8 additions & 0 deletions docs/source/basic_tutorials/launcher.md
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,14 @@ Options:
[env: NGROK_EDGE=]
```
## TOKENIZER_CONFIG_PATH
```shell
--tokenizer-config-path <TOKENIZER_CONFIG_PATH>
The path to the tokenizer config file. This path is used to load the tokenizer configuration which may include a `chat_template`. If not provided, the default config will be used from the model hub
[env: TOKENIZER_CONFIG_PATH=]
```
## ENV
```shell
Expand Down
11 changes: 11 additions & 0 deletions launcher/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -368,6 +368,11 @@ struct Args {
#[clap(long, env)]
ngrok_edge: Option<String>,

/// The path to the tokenizer config file. This path is used to load the tokenizer configuration which may
/// include a `chat_template`. If not provided, the default config will be used from the model hub.
#[clap(long, env)]
tokenizer_config_path: Option<String>,

/// Display a lot of information about your runtime environment
#[clap(long, short, action)]
env: bool,
Expand Down Expand Up @@ -1016,6 +1021,12 @@ fn spawn_webserver(
args.model_id,
];

// Tokenizer config path
if let Some(ref tokenizer_config_path) = args.tokenizer_config_path {
router_args.push("--tokenizer-config-path".to_string());
router_args.push(tokenizer_config_path.to_string());
}

// Model optional max batch total tokens
if let Some(max_batch_total_tokens) = args.max_batch_total_tokens {
router_args.push("--max-batch-total-tokens".to_string());
Expand Down

0 comments on commit d975885

Please sign in to comment.