Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update engines.rst #9

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Update engines.rst #9

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions docs/engines.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ A new engine must implement at least the two abstract methods and set the abstra

With just these three implementations, an engine will be fully functional!

kani comes with a couple additional bases and utilities to help you build engines for models on HuggingFace or with
kani comes with a couple additional bases and utilities to help you build engines for models on Hugging Face or with
an available HTTP API.

Optional Methods
Expand Down Expand Up @@ -59,9 +59,9 @@ Minimally, to use the HTTP client, your subclass should set the ``SERVICE_BASE``
:noindex:
:members:

HuggingFace
Hugging Face
-----------
If your language model backend is available on HuggingFace or is compatible with ``transformers``'
If your language model backend is available on Hugging Face or is compatible with ``transformers``'
``AutoModelForCausalLM`` interface, kani includes a base engine that implements a prediction pipeline.

Instead of having to implement the prediction logic, all you have to do is subclass :class:`.HuggingEngine` and
Expand Down Expand Up @@ -91,11 +91,11 @@ might not be able to fit models with billions of params on your GPU. That's wher
Using FP4 quantization you can expect to reduce up to 8x the model size compared to its native full precision
version.

In this section, we'll show how to load HuggingFace models in FP4.
In this section, we'll show how to load Hugging Face models in FP4.

.. seealso::

We're mostly going to follow the HuggingFace documentation found here:
We're mostly going to follow the Hugging Face documentation found here:
https://huggingface.co/docs/transformers/perf_infer_gpu_one

**Install Dependencies**
Expand Down