Skip to content

Commit cdc2937

Browse files
committed
Make edits to new OpenAI API README section
1 parent 8492db2 commit cdc2937

File tree

1 file changed

+54
-46
lines changed

1 file changed

+54
-46
lines changed

README.md

Lines changed: 54 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -49,54 +49,24 @@ chmod +x llava-v1.5-7b-q4-server.llamafile
4949

5050
**Having trouble? See the "Gotchas" section below.**
5151

52-
### API Quickstart / Alternative to OpenAI API endpoint
53-
54-
Once llamafile server has started, in addition to directly accessing the chat server via <http://127.0.0.1:8080/> a json based API endpoint is also provided.
55-
56-
If you have existing OpenAI based application code relying on OpenAI API endpoint as per [OpenAI Chat Completions API documentation](https://platform.openai.com/docs/api-reference/chat), our API endpoint under base url `http://localhost:8080/v1` is designed to support most OpenAI use cases besides certain OpenAI-specific features such as function calling ( llama.cpp `/completion`-specific features such are `mirostat` are supported.).
57-
58-
For further details on all supported API commands (OpenAI compatible to llamafile specific extention) please refer to [API Endpoint Documentation](llama.cpp/server/README.md#api-endpoints).
59-
60-
#### LLAMAFile Server V1 API Python Example
61-
62-
This shows that you can use existing [OpenAI python package](https://pypi.org/project/openai/) developed by OpenAI because of our compatibility measures.
63-
So most scripts designed for OpenAI will be able to be ported to llamafile with a few changes to base_url and api_key.
64-
65-
<details>
66-
<summary>Python Example Code and Result</summary>
67-
68-
Don't forget to run this command `pip3 install openai` to install the openai package required by this example script. This package is just a simple python wrapper around the openAI's API endpoints.
69-
70-
```python
71-
#!/usr/bin/env python3
72-
from openai import OpenAI
73-
client = OpenAI(
74-
base_url="http://localhost:8080/v1", # "http://<Your api-server IP>:port"
75-
api_key = "sk-no-key-required"
76-
)
77-
completion = client.chat.completions.create(
78-
model="LLaMA_CPP",
79-
messages=[
80-
{"role": "system", "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."},
81-
{"role": "user", "content": "Write a limerick about python exceptions"}
82-
]
83-
)
84-
print(completion.choices[0].message)
85-
```
86-
87-
The above when run would return a python object that may look like below:
88-
89-
```python
90-
ChatCompletionMessage(content='There once was a programmer named Mike\nWho wrote code that would often strike\nAn error would occur\nAnd he\'d shout "Oh no!"\nBut Python\'s exceptions made it all right.', role='assistant', function_call=None, tool_calls=None)
91-
```
92-
93-
</details>
94-
95-
96-
#### LLAMAFile Server V1 API Raw HTTP Request Example
52+
### JSON API Quickstart
53+
54+
When llamafile is started in server mode, in addition to hosting a web
55+
UI chat server at <http://127.0.0.1:8080/>, an [OpenAI
56+
API](https://platform.openai.com/docs/api-reference/chat) chat
57+
completions endpoint is provided too. It's designed to support the most
58+
common OpenAI API use cases, in a way that runs entirely locally. We've
59+
also extended it to include llama.cpp specific features (e.g. mirostat)
60+
that may also be used. For further details on what fields and endpoints
61+
are available, refer to both the [OpenAI
62+
documentation](https://platform.openai.com/docs/api-reference/chat/create)
63+
and the [llamafile server
64+
README](llama.cpp/server/README.md#api-endpoints).
65+
66+
#### Examples
9767

9868
<details>
99-
<summary>Raw HTTP Request Example Command and Result</summary>
69+
<summary>Curl API Client Example</summary>
10070

10171
```shell
10272
curl http://localhost:8080/v1/chat/completions \
@@ -145,6 +115,44 @@ The above when run would return an answer like
145115

146116
</details>
147117

118+
<details>
119+
<summary>Python API Client example</summary>
120+
121+
If you've already developed your software using the [`openai` Python
122+
package](https://pypi.org/project/openai/) (that's published by OpenAI)
123+
then you should be able to port your app to talk to a local llamafile
124+
instead, by making a few changes to `base_url` and `api_key`.
125+
126+
This example assumes you've run `pip3 install openai` to install
127+
OpenAI's client software, which is required by this example. Their
128+
package is just a simple Python wrapper around the OpenAI's API
129+
endpoints.
130+
131+
```python
132+
#!/usr/bin/env python3
133+
from openai import OpenAI
134+
client = OpenAI(
135+
base_url="http://localhost:8080/v1", # "http://<Your api-server IP>:port"
136+
api_key = "sk-no-key-required"
137+
)
138+
completion = client.chat.completions.create(
139+
model="LLaMA_CPP",
140+
messages=[
141+
{"role": "system", "content": "You are ChatGPT, an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."},
142+
{"role": "user", "content": "Write a limerick about python exceptions"}
143+
]
144+
)
145+
print(completion.choices[0].message)
146+
```
147+
148+
The above code will return a Python object like this:
149+
150+
```python
151+
ChatCompletionMessage(content='There once was a programmer named Mike\nWho wrote code that would often strike\nAn error would occur\nAnd he\'d shout "Oh no!"\nBut Python\'s exceptions made it all right.', role='assistant', function_call=None, tool_calls=None)
152+
```
153+
154+
</details>
155+
148156

149157
## Other example llamafiles
150158

0 commit comments

Comments
 (0)