Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add llama.cpp backend #2723

Open
wants to merge 91 commits into
base: main
Choose a base branch
from
Open

Add llama.cpp backend #2723

wants to merge 91 commits into from

Conversation

mfuntowicz
Copy link
Member

@mfuntowicz mfuntowicz commented Nov 4, 2024 •
edited
Loading

This PR is an initial implementation of llama.cpp as potential backend for TGI.

It mostly targets CPU inference in a single/multi stream scheduling fashion, potentially spawning multiple instances of the same model over a non-overlapping subset of the CPU cores.

The current implementation only allows a single request to be running on a working, this constraint will be removed later on.
The current implementation also dupplicate the weights for each worker, this constraint can potentially be removed later on.

@mfuntowicz
feat(llamacpp): initial commit
aa1fcba 
# Conflicts:
#	Cargo.lock
@mfuntowicz
feat(llamacpp): correctly handle CMAKE_BUILD_TYPE for spdlog macros
7d1f8a2
@mfuntowicz
feat(llamacpp): initial end2end build
52d57dc
@mfuntowicz
misc(cmake): add parameter to build specific cuda arch
e4432d3
@mfuntowicz
misc(cmake): wut
fa89d1e
@mfuntowicz
feat(llamacpp): enable cuda
05ad684
@mfuntowicz
feat(backend): correctly load llama.cpp model from llama api and not …
0911076 
…gpt2
@mfuntowicz
feat(backend): tell cmake to build llama-common and link to it
098c669
@mfuntowicz
feat(backend): add some initial decoding steps
45d5a6a
@mfuntowicz
feat(backend): use llama_token as TokenId type
92bb113
@mfuntowicz
feat(backend): minor refactor
d4b5be1
@mfuntowicz
feat(backend): expose frequency and repetition penalties
37faeb3
@mfuntowicz
chore(backend): minor formatting
f9c2486
@mfuntowicz
feat(backend): wip Rust binding
355d8a5
@mfuntowicz
feat(backend): build and link through build.rs
e4d803c
@mfuntowicz
misc(build): handle different lib destination folder lib/lib64
f0859c2
@mfuntowicz
misc(build): refactor build type detection in cmake
179309b
@mfuntowicz
feat(llamacpp): expose number of threads for the backend when constru…
a316c53 
…cting the model
@mfuntowicz
feat(llamacpp): wip explosion
0c1dd0e
@mfuntowicz
misc(offline): link correctly
dbc5b7a
@mfuntowicz
misc(offline): expose more parameters for generate
6115904
@mfuntowicz
feat(backend): entirely rewrite backend
b98c635
@mfuntowicz
misc(offline): update offline tester
6a5f6b0
@mfuntowicz
feat(backend): full rework of the backend internal to safer c++
d52b4c4
@mfuntowicz
misc(offline): match rework
3af2c68
@mfuntowicz
feat(backend): add mapping for ignore_eos_token stopping criteria
f39edc7
@mfuntowicz
feat(backend): add logit parameter in the callback fn
d4aee42
@mfuntowicz
feat(backend): bind incoming request to the server
612f2f9
@mfuntowicz
feat(backend): avoid dropping the boxed stream at the end of the call…
b50dcdd 
…back
@mfuntowicz
feat(backend): somewhat generates the final infer response
3e82f14
@mfuntowicz mfuntowicz marked this pull request as ready for review November 22, 2024 13:47
@mfuntowicz mfuntowicz requested a review from angt November 22, 2024 15:02
Hugoch
Hugoch reviewed Nov 27, 2024
View reviewed changes
Copy link
Member

@Hugoch Hugoch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mfuntowicz ! I can't get a correct generation using the Docker so I think something is wrong with tokenizer. I'll dig deeper later today!

Dockerfile.llamacpp Outdated Show resolved Hide resolved
backends/llamacpp/src/backend.rs Outdated Show resolved Hide resolved
Dockerfile.llamacpp Outdated Show resolved Hide resolved
Dockerfile.llamacpp Show resolved Hide resolved
backends/llamacpp/src/backend.rs Outdated Show resolved Hide resolved
mfuntowicz and others added 17 commits November 28, 2024 09:53
@mfuntowicz @Hugoch
Update Dockerfile.llamacpp as per review
bbe95ca 
Co-authored-by: Hugo Larcher <[email protected]>
@mfuntowicz @Hugoch
Update Dockerfile.llamacpp as per review
d918e6a 
Co-authored-by: Hugo Larcher <[email protected]>
@mfuntowicz
feat(backend): remove core overriding in the Rust backend
274cfce
@mfuntowicz
feat(backend): use the new batch api from llama
8e89793
@mfuntowicz
feat(backend): fix when num_cores_per_instance is equals to zero with…
298367c 
… the size of the generated core allocation
@mfuntowicz
feat(backend): add some test to the backend for core allocation
929a2fc
@mfuntowicz
feat(backend): add guard in case top_k = 0
df72c56
@mfuntowicz
feat(backend): add missing temperature parameter
9d659f1
@mfuntowicz
misc(offline): update model creation as std::shared_ptr
6c5a75b
@mfuntowicz
feat(backend): update llama.cpp to 4215
b1ebc8f
@mfuntowicz
feat(backend): create llama_context_params with default factory
dc6435e
@mfuntowicz
feat(backend): use new batch API to generate tokens
b10eaab
@Hugoch
feat: Fix Cmakelist to allow building on Darwin platform (#2785)
59b0ef3 
* feat: Fix Cmakelist to allow building on Darwin platform
* fix: Fix tokenizer in llama.cpp Dockerfile
@mfuntowicz
feat(backend): correctly link to all libraries
f5c4cee
@mfuntowicz
feat(backend): add mimalloc memory allocator to the container
db41776
@mfuntowicz
feat(backend): better map exception throw on C++ side
c9f6c3a
@mfuntowicz
feat(backend): use c++ defined types for llama.cpp
e0dda9b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Hugoch Hugoch Hugoch left review comments

@co42 co42 Awaiting requested review from co42

@OlivierDehaene OlivierDehaene Awaiting requested review from OlivierDehaene

@angt angt Awaiting requested review from angt

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mfuntowicz @Hugoch