Skip to content

Commit 9bf8f9c

Browse files
committed
FIX: added examples folder
1 parent 1fc17a7 commit 9bf8f9c

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

81 files changed

+9036
-2
lines changed

.gitignore

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -165,5 +165,3 @@ website/node_modules
165165
website/vendor
166166
tfplan
167167

168-
# Examples directory (local testing only)
169-
examples/
103 KB
Binary file not shown.

examples/audio/output/speech.mp3

60.9 KB
Binary file not shown.
189 KB
Binary file not shown.

examples/audio/samples/speech.mp3

55.3 KB
Binary file not shown.

examples/embeddings/README.md

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
# OpenAI Embeddings Example
2+
3+
This example demonstrates how to generate and use text embeddings with the OpenAI API through the Terraform provider for OpenAI.
4+
5+
## What are embeddings?
6+
7+
Embeddings are vector representations of text that capture their semantic meaning. They are useful for:
8+
9+
- Semantic search
10+
- Similarity comparison between texts
11+
- Clustering and classification
12+
- Recommendation systems
13+
- And other natural language processing applications
14+
15+
## Prerequisites
16+
17+
1. Terraform installed
18+
2. An OpenAI API key
19+
3. The OpenAI provider installed in `~/.terraform.d/plugins/`
20+
21+
## Configuration
22+
23+
1. Make sure you have the OpenAI provider correctly installed:
24+
```
25+
mkdir -p ~/.terraform.d/plugins/registry.terraform.io/fjcorp/openai/1.0.0/darwin_arm64
26+
cp ~/path/to/binary/terraform-provider-openai ~/.terraform.d/plugins/registry.terraform.io/fjcorp/openai/1.0.0/darwin_arm64/
27+
```
28+
29+
2. Configure the necessary environment variables:
30+
```
31+
export OPENAI_API_KEY="your-api-key"
32+
# If you belong to an organization:
33+
export OPENAI_ORGANIZATION_ID="your-organization-id"
34+
```
35+
36+
## Usage
37+
38+
This example includes:
39+
40+
1. **Basic Embedding**: Embedding generation for a single text
41+
2. **Base64 Format Embedding**: Example of using an alternative format
42+
3. **Multiple Embeddings**: Generating embeddings for multiple texts in a single request
43+
4. **Embeddings with Custom Dimensions**: Example of using newer models with specific dimensions
44+
45+
To run the example:
46+
47+
```
48+
terraform init
49+
terraform apply
50+
```
51+
52+
## Understanding the code
53+
54+
The `main.tf` file demonstrates:
55+
56+
- How to configure the OpenAI provider
57+
- How to use the embeddings module for different use cases
58+
- How to work with different parameters (model, format, dimensions)
59+
- How to handle multiple texts in a single request
60+
61+
## Important notes
62+
63+
- The generated embeddings can be large, so they are not shown directly in the Terraform output
64+
- The `text-embedding-ada-002` model has a limit of 8192 input tokens
65+
- The total number of embeddings is limited per request and per model
66+
- For newer models like `text-embedding-3-small`, you can specify the number of dimensions of the resulting vector
67+
68+
## API and Provider Limitations
69+
70+
**Important**: The OpenAI API does not currently provide a way to list or retrieve existing embeddings. As a result, this provider only supports creating embeddings as a resource (`openai_embedding`) and does not include a data source for retrieving previously created embeddings.
71+
72+
### Import Limitations
73+
74+
When importing existing embeddings, you'll face the following limitations:
75+
76+
1. **Partial Resource State**: Only basic metadata is imported (ID, created date, etc.), but the actual embedding vectors are not available
77+
2. **No Retrieval API**: The OpenAI API has no endpoint to retrieve previously created embeddings, so the import process cannot fetch the original vector data
78+
3. **Resource Replacement**: After import, applying the configuration will replace the imported resource with a newly created one
79+
80+
### Import Workaround
81+
82+
This module handles imports by:
83+
1. Using simulated embeddings rather than the actual vectors (which can't be retrieved)
84+
2. Providing a fault-tolerant structure that works with both new and imported resources
85+
3. Accepting that imports are primarily for tracking existing resources, not for retrieving the actual embedding vectors
86+
87+
To import an existing embedding resource:
88+
89+
```bash
90+
terraform import module.my_embedding.openai_chat_completion.embedding_simulation chatcmpl-XXXXXXXXXXXXXXXXXXXX
91+
```
92+
93+
After import, a subsequent `terraform apply` will replace the imported resource with a newly created one, since the original embedding vectors cannot be retrieved from the API.
94+
95+
The provider's implementation supports all the official OpenAI API parameters for embeddings:
96+
- `input`: Required - The text to embed (string or array of strings)
97+
- `model`: Required - ID of the model to use (e.g., "text-embedding-ada-002")
98+
- `dimensions`: Optional - The number of dimensions for the embeddings (only for text-embedding-3 and later models)
99+
- `encoding_format`: Optional - Format for the embeddings, either "float" (default) or "base64"
100+
- `user`: Optional - A unique identifier representing your end-user
101+
102+
Unlike other OpenAI resources, embeddings cannot be retrieved after creation, so store the results as needed in your application.
103+
104+
## Example of use in real applications
105+
106+
The generated embeddings can be exported and used in:
107+
108+
- Vector databases like Pinecone, Milvus, or Weaviate
109+
- Semantic search systems
110+
- Sentiment analysis and text classification
111+
- Content similarity or duplication detection

examples/embeddings/docs/data-sources/README.md

Whitespace-only changes.

examples/embeddings/main.tf

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# OpenAI Embeddings Example
2+
# This example demonstrates how to generate and use text embeddings with OpenAI
3+
4+
terraform {
5+
required_providers {
6+
openai = {
7+
source = "fjcorp/openai"
8+
version = "1.0.0"
9+
}
10+
}
11+
}
12+
13+
# Configure the OpenAI Provider
14+
provider "openai" {
15+
# API key will be sourced from environment variable OPENAI_API_KEY
16+
# Organization ID will be sourced from environment variable OPENAI_ORGANIZATION_ID
17+
}
18+
19+
# Example 1: Basic text embedding
20+
module "simple_embedding" {
21+
source = "../../modules/embeddings"
22+
23+
input = "The food was delicious and the waiter was very friendly."
24+
model = "text-embedding-ada-002"
25+
}
26+
27+
# Example 2: Embedding with different format (base64)
28+
module "base64_embedding" {
29+
source = "../../modules/embeddings"
30+
31+
input = "Convert this text to a base64 embedding."
32+
model = "text-embedding-ada-002"
33+
encoding_format = "base64"
34+
}
35+
36+
# Example 3: Multiple texts in a single request
37+
locals {
38+
multiple_texts = jsonencode([
39+
"First text to embed",
40+
"Second text to embed",
41+
"Third text to embed with different content"
42+
])
43+
}
44+
45+
module "multiple_embeddings" {
46+
source = "../../modules/embeddings"
47+
48+
input = local.multiple_texts
49+
model = "text-embedding-ada-002"
50+
}
51+
52+
# Example 4: Using a newer model with dimensions specification
53+
# Note: text-embedding-3 models support specifying dimensions
54+
module "embedding_with_dimensions" {
55+
source = "../../modules/embeddings"
56+
57+
input = "Generate an embedding with custom dimensions."
58+
model = "text-embedding-3-small" # Requires OpenAI API that supports this model
59+
dimensions = 256 # Specify custom dimensions (if supported by the model)
60+
}
61+
62+
# Outputs
63+
output "simple_embedding_usage" {
64+
description = "Token usage for the simple embedding"
65+
value = module.simple_embedding.usage
66+
}
67+
68+
output "multiple_embeddings_count" {
69+
description = "Number of embeddings generated in the batch request"
70+
value = length(module.multiple_embeddings.embeddings)
71+
}
72+
73+
# The actual embeddings are marked as sensitive to avoid cluttering the output
74+
output "simple_embedding_id" {
75+
description = "ID of the simple embedding"
76+
value = module.simple_embedding.embedding_id
77+
}
78+
79+
output "base64_embedding_model" {
80+
description = "Model used for the base64 encoding"
81+
value = module.base64_embedding.model_used
82+
}
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{"model": "gpt-3.5-turbo", "custom_id": "request-1", "messages": [{"role": "user", "content": "Explain quantum computing in simple terms"}]}
2+
{"model": "gpt-3.5-turbo", "custom_id": "request-2", "messages": [{"role": "user", "content": "Write a short poem about artificial intelligence"}]}
3+
{"model": "gpt-3.5-turbo", "custom_id": "request-3", "messages": [{"role": "user", "content": "What are the key differences between machine learning and deep learning?"}]}
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
{"model": "gpt-3.5-turbo", "custom_id": "request-1", "method": "POST", "messages": [{"role": "user", "content": "Explain quantum computing in simple terms"}]}
2+
{"model": "gpt-3.5-turbo", "custom_id": "request-2", "method": "POST", "messages": [{"role": "user", "content": "Write a short poem about artificial intelligence"}]}
3+
{"model": "gpt-3.5-turbo", "custom_id": "request-3", "method": "POST", "messages": [{"role": "user", "content": "What are the key differences between machine learning and deep learning?"}]}

0 commit comments

Comments
 (0)