-
Notifications
You must be signed in to change notification settings - Fork 13
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
1 parent
cb978f7
commit 97dae50
Showing
1 changed file
with
13 additions
and
8 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,14 +9,25 @@ Features: | |
|
||
- ✅ Easy to deploy: single binary with no dependencies | ||
- ✅ Single yaml configuration file | ||
- ✅ Automatic switching between models | ||
- ✅ Full control over llama.cpp server settings per model | ||
- ✅ On-demand model switching | ||
- ✅ Full control over server settings per model | ||
- ✅ OpenAI API support (`v1/completions` and `v1/chat/completions`) | ||
- ✅ Multiple GPU support | ||
- ✅ Run multiple models at once with `profiles` | ||
- ✅ Remote log monitoring at `/log` | ||
- ✅ Automatic unloading of models from GPUs after timeout | ||
|
||
## Releases | ||
|
||
Builds for Linux and OSX are available on the [Releases](https://github.com/mostlygeek/llama-swap/releases) page. | ||
|
||
### Building from source | ||
|
||
1. Install golang for your system | ||
1. `git clone [email protected]:mostlygeek/llama-swap.git` | ||
1. `make clean all` | ||
1. Binaries will be in `build/` subdirectory | ||
|
||
## config.yaml | ||
|
||
llama-swap's configuration is purposefully simple. | ||
|
@@ -126,9 +137,3 @@ StartLimitInterval=30 | |
[Install] | ||
WantedBy=multi-user.target | ||
``` | ||
## Building from Source | ||
1. Install golang for your system | ||
1. run `make clean all` | ||
1. binaries will be built into `build/` directory |