-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore: Follow-up Documentation cleanup #159
Changes from 62 commits
7239bbb
f89fc29
971883e
8072128
f39f044
9028c12
e0f2033
df50cd1
7390dbe
8506bd6
361940b
ef23a7c
d62d62d
98fa3b4
f1991b1
864f041
7b94a23
9333645
63fd72c
94c3d8a
4c8ce4f
87a2ac0
124a0d8
54f7815
0f34a2d
d5856df
45b547e
2e29d74
8fd9186
76f5dd3
408b0a1
0016f88
756b5df
41c0b84
129bdeb
0a2a768
728abe2
afb3a95
267a6f7
53b14e2
cd54014
b379d9e
40e7409
688af41
9e26d5a
d7f9522
69afd57
e24d8ad
7af7b3b
508fee5
697e2c9
1425977
5d40c54
ecf5c01
831b355
5358ad1
bce751d
57c02a0
37f23a7
8188acc
05d15b8
59cb45a
1838278
9eff8b0
9926785
3a70dc7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,14 +4,16 @@ This package provides LangChain model clients built on top of the foundation mod | |
|
||
## Table of Contents | ||
|
||
1. [Installation](#installation) | ||
2. [Prerequisites](#prerequisites) | ||
3. [Usage](#usage) | ||
- [Client Initialization](#client-initialization) | ||
- [Chat Client](#chat-client) | ||
- [Embedding Client](#embedding-client) | ||
4. [Support, Feedback, Contribution](#support-feedback-contribution) | ||
5. [License](#license) | ||
- [Installation](#installation) | ||
- [Prerequisites](#prerequisites) | ||
- [Relationship between Models and Deployment ID](#relationship-between-models-and-deployment-id) | ||
- [Usage](#usage) | ||
- [Client Initialization](#client-initialization) | ||
- [Chat Client](#chat-client) | ||
- [Embedding Client](#embedding-client) | ||
- [Local Testing](#local-testing) | ||
- [Support, Feedback, Contribution](#support-feedback-contribution) | ||
- [License](#license) | ||
|
||
## Installation | ||
|
||
|
@@ -24,10 +26,22 @@ $ npm install @sap-ai-sdk/langchain | |
- [Enable the AI Core service in SAP BTP](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/initial-setup). | ||
- Bind the service to your application. | ||
- Ensure the project is configured with Node.js v20 or higher, along with native ESM support. | ||
- For testing your application locally: | ||
- Download a service key for your AI Core service instance. | ||
- Create a `.env` file in the root of your directory. | ||
- Add an entry `AICORE_SERVICE_KEY='<content-of-service-key>'`. | ||
- A deployed model is available in SAP Generative AI hub. | ||
- Use the [`DeploymentApi`](https://github.com/SAP/ai-sdk-js/blob/main/packages/ai-api/README.md#create-a-deployment) from `@sap-ai-sdk/ai-api` to deploy a model to SAP generative AI hub. | ||
For more information, see [here](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-deployment-for-generative-ai-model-in-sap-ai-core). | ||
- Once a deployment is complete, the model can be accessed via the `deploymentUrl`. | ||
|
||
## Relationship between Models and Deployment ID | ||
|
||
SAP AI Core manages access to generative AI models through the global AI scenario `foundation-models`. | ||
Creating a deployment for a model requires access to this scenario. | ||
|
||
Each model, model version, and resource group allows for a one-time deployment. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same as before, a bit confusing to me. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same as above. |
||
After deployment completion, the response includes a `deploymentUrl` and an `id`, which is the deployment ID. For more information, see [here](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-deployment-for-generative-ai-model-in-sap-ai-core). | ||
|
||
[Resource groups](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/resource-groups?q=resource+group) represent a virtual collection of related resources within the scope of one SAP AI Core tenant. | ||
|
||
Consequently, each deployment ID and resource group uniquely map to a combination of model and model version within the `foundation-models` scenario. | ||
|
||
## Usage | ||
|
||
|
@@ -60,6 +74,9 @@ const chatClient = new AzureOpenAiChatClient({ | |
}); | ||
``` | ||
|
||
**Do not pass a `deployment ID` to initialize the client.** | ||
For the LangChain model clients, initialization is done using the model name, model version and resource group. | ||
|
||
An important note is that LangChain clients by default attempt 6 retries with exponential backoff in case of a failure. | ||
Especially in testing environments you might want to reduce this number to speed up the process: | ||
|
||
|
@@ -155,6 +172,10 @@ const vectorStore = await MemoryVectorStore.fromDocuments( | |
const retriever = vectorStore.asRetriever(); | ||
``` | ||
|
||
## Local Testing | ||
|
||
For local testing instructions, refer to this [section](https://github.com/SAP/ai-sdk-js/blob/main/README.md#local-testing). | ||
|
||
## Support, Feedback, Contribution | ||
|
||
This project is open to feature requests/suggestions, bug reports etc. via [GitHub issues](https://github.com/SAP/ai-sdk-js/issues). | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -2,16 +2,18 @@ | |
|
||
This package incorporates generative AI orchestration capabilities into your AI activities in SAP AI Core and SAP AI Launchpad. | ||
|
||
### Table of Contents | ||
## Table of Contents | ||
|
||
- [Installation](#installation) | ||
- [Prerequisites](#prerequisites) | ||
- [Prerequisites](#prerequisites) | ||
- [Orchestration Service](#orchestration-service) | ||
- [Relationship between Orchestration and Resource Groups](#relationship-between-orchestration-and-resource-groups) | ||
- [Usage](#usage) | ||
- [Templating](#templating) | ||
- [Token Usage](#token-usage) | ||
- [Content Filtering](#content-filtering) | ||
- [Data Masking](#data-masking) | ||
- [Data Masking](#data-masking) | ||
- [Using Resource Groups](#using-resource-groups) | ||
- [Local Testing](#local-testing) | ||
- [Support, Feedback, Contribution](#support-feedback-contribution) | ||
- [License](#license) | ||
|
||
|
@@ -21,12 +23,13 @@ This package incorporates generative AI orchestration capabilities into your AI | |
$ npm install @sap-ai-sdk/orchestration | ||
``` | ||
|
||
### Prerequisites | ||
## Prerequisites | ||
|
||
- [Enable the AI Core service in SAP BTP](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/initial-setup). | ||
- Project configured with Node.js v20 or higher and native ESM support enabled. | ||
- An orchestration deployment is running. | ||
- Use the [`DeploymentApi`](../ai-api/README.md#deploymentapi) from `@sap-ai-sdk/ai-api` to create a deployment for orchestration to the SAP generative AI hub. For more information, see [here](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-deployment-for-orchestration). | ||
- Use the [`DeploymentApi`](https://github.com/SAP/ai-sdk-js/blob/main/packages/ai-api/README.md#create-a-deployment) from `@sap-ai-sdk/ai-api` to create a deployment for orchestration to the SAP generative AI hub. | ||
For more information, see [here](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/create-deployment-for-orchestration). | ||
- Once a deployment is complete, the orchestration service can be accessed via the `deploymentUrl`. | ||
|
||
## Orchestration Service | ||
|
@@ -38,6 +41,16 @@ The orchestration service provides essential features like templating and conten | |
|
||
Find more details about orchestration workflow [here](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/orchestration-workflow). | ||
|
||
## Relationship between Orchestration and Resource Groups | ||
|
||
SAP AI Core manages access to orchestration of generative AI models through the global AI scenario `orchestration`. | ||
Creating a deployment for enabling orchestration capabilities requires access to this scenario. | ||
|
||
[Resource groups](https://help.sap.com/docs/sap-ai-core/sap-ai-core-service-guide/resource-groups?q=resource+group) represent a virtual collection of related resources within the scope of one SAP AI Core tenant. | ||
Each resource group allows for a one-time orchestration deployment. | ||
|
||
Consequently, each orchestration deployment uniquely maps to a resource group within the `orchestration` scenario. | ||
deekshas8 marked this conversation as resolved.
Show resolved
Hide resolved
Comment on lines
+45
to
+53
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [pp] I would suggest we extract this section into a separate file as it has been repeated many times. It will be very hard to maintain so many packages. At least we should do this definitely in the future. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As, the deployment requirements change between orchestration and foundation-models. I would prefer to write about it in the dedicated client document instead of a separate file, as this would be more confusing. |
||
|
||
## Usage | ||
|
||
Leverage the orchestration service capabilities by using the orchestration client. | ||
|
@@ -99,6 +112,29 @@ const response = await orchestrationClient.chatCompletion( | |
const responseContent = response.getContent(); | ||
``` | ||
|
||
`getContent()` is a convenience method that parses the response and returns the model's output as a string. | ||
|
||
To retrieve the `finish_reason` for stopping the chat completion request, use the convenience method `getFinishReason()`: | ||
|
||
```ts | ||
deekshas8 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
const finishReason = response.getFinishReason(); | ||
``` | ||
|
||
Use the `getTokenUsage()` convenience method to retrieve the token usage details of the chat completion request: | ||
|
||
deekshas8 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
```ts | ||
const tokenUsage = response.getTokenUsage(); | ||
|
||
console.log( | ||
`Total tokens consumed by the request: ${tokenUsage.total_tokens}\n` + | ||
`Input prompt tokens consumed: ${tokenUsage.prompt_tokens}\n` + | ||
`Output text completion tokens consumed: ${tokenUsage.completion_tokens}\n` | ||
); | ||
``` | ||
|
||
deekshas8 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
#### Passing a Message History | ||
|
||
It is possible to provide a history of a conversation to the model. | ||
Use the following snippet to send a chat completion request with history and a system message: | ||
|
||
|
@@ -136,22 +172,6 @@ const response = await orchestrationClient.chatCompletion({ | |
const responseContent = response.getContent(); | ||
``` | ||
|
||
#### Token Usage | ||
|
||
To retrieve the token usage details of the orchestration request, use the following snippet: | ||
|
||
```ts | ||
const tokenUsage = response.getTokenUsage(); | ||
|
||
logger.info( | ||
`Total tokens consumed by the request: ${tokenUsage.total_tokens}\n` + | ||
`Input prompt tokens consumed: ${tokenUsage.prompt_tokens}\n` + | ||
`Output text completion tokens consumed: ${tokenUsage.completion_tokens}\n` | ||
); | ||
``` | ||
|
||
Remember to initialize a logger before using it. | ||
|
||
### Content Filtering | ||
|
||
Use the orchestration client with filtering to restrict content that is passed to and received from a generative AI model. | ||
|
@@ -189,11 +209,21 @@ try { | |
} | ||
``` | ||
|
||
Both `chatCompletion()` and `getContent()` methods can throw errors. | ||
|
||
- **Axios Errors**: | ||
When the chat completion request fails with a `400` status code, the caught error will be an `Axios` error. The property `error.response.data.message` may provide additional details about the failure's cause. | ||
deekshas8 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
- **Output Content Filtered**: | ||
The method `getContent()` can throw an error if the output filter filters the model output. This situation can occur even if the chat completion request succeeds with a `200` status code. The `error.message` property indicates whether the output was filtered. | ||
deekshas8 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Therefore, handle errors appropriately to ensure meaningful feedback for both types of errors. | ||
|
||
`buildAzureContentFilter()` is a convenience function that creates an Azure content filter configuration based on the provided inputs. | ||
The Azure content filter supports four categories: `Hate`, `Violence`, `Sexual`, and `SelfHarm`. | ||
Each category can be configured with severity levels of 0, 2, 4, or 6. | ||
|
||
#### Data Masking | ||
### Data Masking | ||
|
||
You can anonymize or pseudonomize the prompt using the data masking capabilities of the orchestration service. | ||
|
||
|
@@ -229,6 +259,31 @@ const response = await orchestrationClient.chatCompletion({ | |
return response.getContent(); | ||
``` | ||
|
||
### Using Resource Groups | ||
|
||
The resource group can be used as an additional parameter to pick the right orchestration deployment. | ||
|
||
```ts | ||
const orchestrationClient = new OrchestrationClient( | ||
{ | ||
llm: { | ||
model_name: 'gpt-4-32k', | ||
model_params: { max_tokens: 50, temperature: 0.1 } | ||
}, | ||
templating: { | ||
template: [{ role: 'user', content: 'What is my name?' }] | ||
} | ||
}, | ||
{ resourceGroup: 'rg1234' } | ||
); | ||
``` | ||
|
||
The relationship between orchestration and resource groups is explained in this [section](#relationship-between-orchestration-and-resource-groups). | ||
|
||
## Local Testing | ||
|
||
For local testing instructions, refer to this [section](https://github.com/SAP/ai-sdk-js/blob/main/README.md#local-testing). | ||
|
||
## Support, Feedback, Contribution | ||
|
||
Contribution and feedback are encouraged and always welcome. For more information about how to contribute, the project structure, as well as additional contribution information, see our [Contribution Guidelines](https://github.com/SAP/ai-sdk-js/blob/main/CONTRIBUTING.md). | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Subtle differences between the two, so I will use the original sentence, as it talks about relationship rather than what is required by the API.