Skip to content

Commit

Permalink
Add vector database description
Browse files Browse the repository at this point in the history
  • Loading branch information
Adriankhl committed May 21, 2024
1 parent d204531 commit 12838c8
Show file tree
Hide file tree
Showing 4 changed files with 17 additions and 11 deletions.
14 changes: 11 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ A template/demo to show how to use the [godot-llm](https://github.com/Adriankhl/
## Image to text
![](./media/demo-image-to-text.gif)

## Vector database
![](./media/demo-vector-database.gif)

# How to use
You may simply download the Android apk in the release page to try it out for Android devices.

Expand All @@ -21,7 +24,7 @@ For Windows/Linux:

## Download models
* Text generation recommendation: [Meta-Llama-3-8B-Instruct-Q5_K_M.gguf](https://huggingface.co/lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF/tree/main) or [Phi-3-mini-4k-instruct-Q2_K.gguf](https://huggingface.co/bartowski/Phi-3-mini-4k-instruct-GGUF/tree/main) for low-end devices such as Android phone
* Similarity recommendation: [mxbai-embed-large-v1.Q5_K_M.gguf](https://huggingface.co/ChristianAzinn/mxbai-embed-large-v1-gguf/tree/main)
* Similarity/Vector database recommendation: [mxbai-embed-large-v1.Q5_K_M.gguf](https://huggingface.co/ChristianAzinn/mxbai-embed-large-v1-gguf/tree/main)
* Image to text recommendation: [llava-phi-3-mini-int4.gguf](https://huggingface.co/xtuner/llava-phi-3-mini-gguf/tree/main), also download the `llava-phi-3-mini-mmproj-f16.gguf` from the same page

Folder and file structure:
Expand All @@ -36,6 +39,8 @@ Folder and file structure:
│   │   ├── libgodot_llm.windows.debug.amd64.dll
│   │   └── libgodot_llm.windows.release.amd64.dll
│   └── godot_llm.gdextension
├── db.gd
├── db.tscn
├── icon.svg
├── icon.svg.import
├── image_to_text.gd
Expand Down Expand Up @@ -69,7 +74,7 @@ Just edit the prompt and click `Generate`. Click `Stop` to stop the generation.

You choose `None` schema or `Person` schema, if you choose `None` schema, the model will generate the information of a character with Json format.

### `Instruct` mode
### `Instruct` mode (currently this is disabled)
Click `Start` first with empty prompt, then modify the prompt and click `Continue` when it is available to talk to the AI. This is an interactive mode. Whenever you see the `Continue` button is available, either input something in the prompt and click `Continue` to send the input, or simply click `Continue` to let the AI talk itself.

### `Interactive` mode
Expand All @@ -79,4 +84,7 @@ Similar to the instruct mode, except that it relies on a good initial prompt and
Input prompts to compute the `Embedding`, or input two prompts to calculate their `Similarity`

## Image to Text
Select an `Image` from your device. Alternative, click `From View` to capture your game screen as the image. You can `Compute Base64` encoding for your image, or decode a base64 string back to `Compute Image`. Edit the prompt, and generate text from `Image`, `Base64`, or `View` (your game screen).
Select an `Image` from your device. Alternative, click `From View` to capture your game screen as the image. You can `Compute Base64` encoding for your image, or decode a base64 string back to `Compute Image`. Edit the prompt, and generate text from `Image`, `Base64`, or `View` (your game screen).

## Vector Database
Click `Metadata: id, year` to define the metadata of the tables, then click `Create tables` to create a some tables for embedding storage and retrieval. Click `Store text` to split the document into chunks and store into the database, you may also click `Split text` to see the chunks. Click `Retrieve` to retrive three most similar text chunks with the upper left prompt, which satisfy the sql-where-clause in the middle right. You may also turn on "Verbose stdout", modify the upper prompt to a valid sql statement, and click execute to run the sql statement directly, you will see the result in godot log.
4 changes: 1 addition & 3 deletions db.gd
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ func _on_drop_table_button_pressed():

func _on_split_text_button_pressed():
var array = $LlmDB.split_text($Document.text)
$Output.clear()
$Output.text = ""
for s in array:
$Output.text += s + "\n\n"

Expand Down Expand Up @@ -67,7 +67,6 @@ func _on_llm_db_retrieve_similar_texts_finished(array):


func _on_back_button_pressed():
$Llama.stop_generate_text()
get_tree().change_scene_to_file("res://main.tscn")


Expand All @@ -82,4 +81,3 @@ func _on_model_chooser_file_selected(path):

func _on_execute_button_pressed():
$LlmDB.execute($Prompt.text)
print($LlmDB.has_id("Document2022", "llm_table_meta"))
10 changes: 5 additions & 5 deletions db.tscn
Original file line number Diff line number Diff line change
Expand Up @@ -52,12 +52,12 @@ grow_vertical = 0
theme_override_styles/normal = SubResource("StyleBoxFlat_m03tj")

[node name="MetaButton" type="Button" parent="."]
offset_left = 76.0
offset_top = 49.0
offset_right = 237.0
offset_bottom = 80.0
offset_left = 155.0
offset_top = 48.0
offset_right = 445.0
offset_bottom = 101.0
theme_override_font_sizes/font_size = 32
text = "Meta data schema: id, year"
text = "Metadata: id, year"

[node name="CreateTableButton" type="Button" parent="."]
offset_left = 69.0
Expand Down
Binary file added media/demo-vector-database.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 12838c8

Please sign in to comment.