feat: tokenization endpoint

[shraddhazpy]

Description

This PR is a step towards #1649

Notes for Reviewers

Signed commits

• Yes, I signed my commits.

[netlify]

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	fa0f048
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/66fcce2255dd240008decfac
😎 Deploy Preview	https://deploy-preview-3710--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[shraddhazpy]

@mudler Guessing that this would need work on the cpp side as well to get the tokenization going there too.

[mudler]

@mudler Guessing that this would need work on the cpp side as well to get the tokenization going there too.

Correct, we basically need to expose

LocalAI/backend/cpp/llama/grpc-server.cpp

Line 549 in 44bdaca

std::vector<llama_token> tokenize(const json & json_prompt, bool add_bos) const

as a gRPC method, but it would make sense also to do it afterwards 👍 - thanks for the PR!

[mudler]

@shraddhazpy I need the commits signed off (you can do that with git rebase origin/master --signoff) in order to merge the PR

[mudler]

Looking good!