Documentation of server command line parameters.

[arthurwolf]

I run python3 -m llama_cpp.server in order to call the API from my scripts.

I'd like to implement prompt caching (like I can do in llama-cpp), but the command line options that work for llama-cpp server don't work for this project.

I search the docs and couldn't find docs on the command line options that would work.

After an error trying random command line options, I did get this output on the command line:

/home/arthur/.local/lib/python3.11/site-packages/pydantic/_internal/_fields.py:127: UserWarning: Field "model_alias" has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ('settings_',)`.
  warnings.warn(
usage: __main__.py [-h] [--model MODEL] [--model_alias MODEL_ALIAS] [--n_ctx N_CTX] [--n_gpu_layers N_GPU_LAYERS] [--tensor_split TENSOR_SPLIT]
                   [--rope_freq_base ROPE_FREQ_BASE] [--rope_freq_scale ROPE_FREQ_SCALE] [--seed SEED] [--n_batch N_BATCH] [--n_threads N_THREADS]
                   [--f16_kv F16_KV] [--use_mlock USE_MLOCK] [--use_mmap USE_MMAP] [--embedding EMBEDDING] [--low_vram LOW_VRAM]
                   [--last_n_tokens_size LAST_N_TOKENS_SIZE] [--logits_all LOGITS_ALL] [--cache CACHE] [--cache_type CACHE_TYPE] [--cache_size CACHE_SIZE]
                   [--vocab_only VOCAB_ONLY] [--verbose VERBOSE] [--host HOST] [--port PORT] [--interrupt_requests INTERRUPT_REQUESTS] [--n_gqa N_GQA]
                   [--rms_norm_eps RMS_NORM_EPS] [--mul_mat_q MUL_MAT_Q]

From which I can see these look like what I'm looking for:

[--cache CACHE] [--cache_type CACHE_TYPE] [--cache_size CACHE_SIZE]

However:

1. I have no idea what the format for CACHE, CACHE_TYPE and CACHE_SIZE or, or the precise meaning/effect of each option.
2. I would be very interrested in knowing what the othe options mean also.

Is there any documentation anywhere of what these mean/how to use them?

( following the exact same format/names as llamma cpp might be a good idea wherever possible btw, it would have enabled me to get this to work without bothering you, as using the llama cpp formats/options is the first thing I tried)..

Thanks a lot for any possible help.

Best regards.

[gjmulder]

https://abetlen.github.io/llama-cpp-python/

[arthurwolf]

https://abetlen.github.io/llama-cpp-python/

Thanks, @gjmulder that's interesting, but unless I'm missing something, it doesn't actually answer my questions about the parameters for prompt caching over the command line.

[Ph0rk0z]

They're supposed to be same as l.cpp params.

[arthurwolf]

They're supposed to be same as l.cpp params.

Yes, but they're not...

Is that an error/bug?

Can this be expected to be fixed with time?

Should I file a bug report?

[Ph0rk0z]

Cache is prompt cache I think and type is disk or ram and size is well..size.

[arthurwolf]

Thanks for the info. What's the format for the size?

[abetlen]

@arthurwolf it's just number of bytes, no custom parsing. Also for the cli passing --help should give a more information.