Text-to-SQL proof of concept #5788

ad-elias · 2024-06-09T19:13:09Z

Added:

An "Ask AI" command to the command menu.
A simple GraphQL resolver that converts the user's question into a relevant SQL query using an LLM, runs the query, and returns the result.

No security concerns have been addressed, this is only a proof-of-concept and not intended to be enabled in production.

All changes are behind a feature flag called IS_ASK_AI_ENABLED.

github-actions · 2024-06-09T19:14:02Z

	Warnings
⚠️	Changes were made to the environment variables, but not to the documentation - Please review your changes and check if a change needs to be documented!

Welcome!

Hello there, congrats on your first PR! We're excited to have you contributing to this project.
By submitting your Pull Request, you acknowledge that you agree with the terms of our Contributor License Agreement.

TODOs/FIXMEs:

``// TODO

Icon: IconSparkles,``: packages/twenty-front/src/modules/command-menu/components/CommandMenu.tsx

Generated by 🚫 dangerJS against 8100e21

greptile-apps

PR Summary

Added "Ask AI" command to command menu
Introduced Text-to-SQL GraphQL resolver and service
Added langchain dependency and updated typeorm
Integrated new feature flag IS_ASK_AI_ENABLED
Updated core engine module to include TextToSQLModule

packages/twenty-front/src/generated/graphql.tsx

packages/twenty-server/src/engine/core-modules/text-to-sql/text-to-sql.resolver.ts

packages/twenty-server/src/engine/core-modules/text-to-sql/text-to-sql.service.ts

FelixMalfait · 2024-06-09T19:59:56Z

😍 amazing! Looking forward to reviewing this first proof of concept!

FelixMalfait

That's great work! It's 100% in the right direction.
I tried it locally and the results seemed to be disappointing (only got Error executing raw query for workspace ....: syntax error at or near ... for every query I tried), any idea why? I didn't dig into it / didn't even print the intermediate sql result to investigate yet

package.json

packages/twenty-front/src/modules/command-menu/components/CommandMenu.tsx

packages/twenty-server/src/engine/core-modules/text-to-sql/text-to-sql.module.ts

packages/twenty-server/src/engine/core-modules/text-to-sql/text-to-sql.resolver.ts

FelixMalfait · 2024-06-09T20:33:16Z

packages/twenty-server/src/engine/core-modules/text-to-sql/text-to-sql.service.ts

+    const workspaceSchemaName =
+      this.workspaceDataSourceService.getSchemaName(workspaceId);
+
+    const workspaceDataSource =


Maybe we should hook into WorkspaceQueryRunner service which already has similar code?
As of now WorkspaceQueryRunner forces query execution through graphql.resolve but maybe refactoring a bit we could rely on SET search_path etc and not duplicate that logic? Not 100% sure I understand every single line but it seems similar to what you did here

greptile-apps

PR Summary

(updates since last review)

Introduced 'Ask AI' feature to convert user questions into SQL queries using LLM
Added new components and hooks for 'Ask AI' in the right drawer
Updated command menu to include 'Ask AI' command
Modified existing components to support new feature
Added new environment variables and dependencies for OpenAI and Langfuse integration

...wenty-server/src/engine/api/graphql/workspace-query-runner/workspace-query-runner.service.ts

packages/twenty-server/src/engine/core-modules/ask-ai/ask-ai.resolver.ts

packages/twenty-server/src/engine/core-modules/ask-ai/ask-ai.service.ts

packages/twenty-server/.env.example

...ges/twenty-front/src/modules/activities/ask-ai/right-drawer/components/RightDrawerAIChat.tsx

FelixMalfait · 2024-06-16T09:08:44Z

packages/twenty-server/src/engine/core-modules/ask-ai/ask-ai.service.ts

@@ -0,0 +1,95 @@
+import { Injectable } from '@nestjs/common';
+
+import { ChatOpenAI } from '@langchain/openai';


Do you use https://platform.openai.com/docs/guides/text-generation/json-mode and https://platform.openai.com/docs/guides/function-calling? You probably should?

I understand why we would eventually not want to use function calling eventually to avoid coupling too much with Open AI? But JSON mode can only be positive?

I'll check this out if non-SQL repsonses from the LLM keep being an issue after iterating the prompt!

FelixMalfait · 2024-06-16T09:09:54Z

packages/twenty-server/src/engine/core-modules/ask-ai/ask-ai.service.ts

+
+    const sqlQuery = await sqlQueryGeneratorChain.invoke(
+      {
+        schema: await db.getTableInfo(),


As discussed via Discord you can look into workspaceCacheVersion ; graphql and sql schema can both be synced and used the same version / invalidation

FelixMalfait

Great work! Some minor comments that could be addressed later, except for getRecordMetadataById where I feel you're headed in a wrong direction (although I might have not understood it very well)

...ges/twenty-front/src/modules/activities/ask-ai/right-drawer/components/RightDrawerAIChat.tsx

FelixMalfait · 2024-06-23T09:01:45Z

...ges/twenty-front/src/modules/activities/ask-ai/right-drawer/components/RightDrawerAIChat.tsx

+                    recordMetadataById={data.getAskAI.recordMetadataById}
+                  />
+                ) : (
+                  'Invalid SQL query.'


Probably when query is invalid we should let the user have an option to view the error.
E.g. I tried this and it was frustrating not to be able to see the error as it wasn't obvious to me (v2 to add to roadmap: let AI try to fix the error)

packages/twenty-server/src/engine/core-modules/ask-ai/ask-ai.service.ts

packages/twenty-server/src/engine/integrations/environment/environment-variables.ts

FelixMalfait · 2024-06-23T09:38:27Z

packages/twenty-server/src/engine/integrations/llm-prompt-template/drivers/file.driver.ts

+    promptTemplateName: string,
+  ): Promise<PromptTemplate<PromptTemplateInput<T>, any>> {
+    const filePath = join(
+      resolveAbsolutePath('.llm-prompt-templates'),


I understand why you've made this choice and it's a good choice. But subjectively I would prefer to keep the root clean if you just put it in within /llm-prompt-template/ folder. Maybe create folders within driver (/file/file.driver.ts, /langfuse/langfuse.driver.ts ; and put it in /llm-prompt-template/drivers/file/templates/*.txt). Thanks

I moved the prompt to a variable and deleted LLMPromptTemplateDriver for now.

It's probably not as useful as I thought to store the prompt template in a separate file or in Langfuse, because the most frequently changed and important parts of the prompt will be dynamically generated (not editable in Langfuse).

...src/engine/integrations/llm-prompt-template/interfaces/llm-prompt-template-name.interface.ts

packages/twenty-server/src/engine/core-modules/ask-ai/ask-ai.service.ts

FelixMalfait · 2024-06-23T20:31:59Z

packages/twenty-server/src/engine/core-modules/ask-ai/ask-ai.service.ts

+    return recordMetadataById;
+  }
+
+  async query(


One challenge is pagination / how to handle a large number of results. It feels like there's a divergence between UI and CSV/download

Could be for exemple, show count and limit to 30 results in view (because we enrich the data which is costly and poorly optimized). When downloading CSV view all results because we don't enrich?

ad-elias added 9 commits June 5, 2024 12:35

Create text-to-SQL resolver

8b78ee0

Simple text-to-SQL request from frontend

8299c0d

Return query and result separately

38b7a3c

Generate text-to-SQL hooks

8105e1e

Simple Ask AI frontend

5bdea2c

Add IsAskAiEnabled feature flag to the backend

0f357f4

Move feature flag check to resolver

7df681b

IsAskAiEnabled frontend feature flag

00f9822

Change command icon

cfc50b3

greptile-apps bot reviewed Jun 9, 2024

View reviewed changes

FelixMalfait reviewed Jun 9, 2024

View reviewed changes

FelixMalfait added the PR: awaiting author label Jun 9, 2024

ad-elias added 10 commits June 10, 2024 12:14

Disable sample rows in schema prompt

a36e7a0

Add OPENAI_API_KEY to .env.example

3441142

Add Langfuse tracing

a0e64a9

Add Langfuse prompt management

7bb0a96

Rename textToSQL to askAI

cc76b10

Remove default question

1957eb4

Refactor

9d93445

Add metadata to tracing

4e4ed56

Create RightDrawer for Ask AI

1563b10

Ask AI result view styling

842cae8

greptile-apps bot reviewed Jun 14, 2024

View reviewed changes

FelixMalfait reviewed Jun 16, 2024

View reviewed changes

packages/twenty-server/.env.example Outdated Show resolved Hide resolved

FelixMalfait reviewed Jun 16, 2024

View reviewed changes

packages/twenty-server/.env.example Outdated Show resolved Hide resolved

FelixMalfait reviewed Jun 16, 2024

View reviewed changes

...ges/twenty-front/src/modules/activities/ask-ai/right-drawer/components/RightDrawerAIChat.tsx Outdated Show resolved Hide resolved

FelixMalfait reviewed Jun 16, 2024

View reviewed changes

ad-elias added 14 commits June 17, 2024 16:59

Very simple error handling

f945a08

Move AI env var docs to self-hosting-var.mdx

3938aaa

Create Langfuse prompt template driver

0f55607

Create file prompt template driver

3c080ef

Create OpenAI ChatModel driver

6ec164a

Use EnvironmentService

c065401

Create LLM Tracing drivers

bbb7b9f

Very simple sql query result table

86d9f04

Merge branch 'main' into feature/text-to-sql

c4bc727

Fix warnings

bd16a84

Add env var documentation

7463f4a

Remove logging

54cfa9a

Fix loading state

d826382

Record links in table WiP

31e6f24

FelixMalfait reviewed Jun 23, 2024

View reviewed changes

Use theme spacing

390f368

FelixMalfait reviewed Jun 23, 2024

View reviewed changes

packages/twenty-server/src/engine/core-modules/ask-ai/ask-ai.service.ts Outdated Show resolved Hide resolved

FelixMalfait reviewed Jun 23, 2024

View reviewed changes

ad-elias added 12 commits June 24, 2024 01:36

Use JSON mode

4f8fd27

Store LLM prompts in code

cb2efc6

Rename to RecordDisplayDataById

2b62e57

Manual schema gen to remove unnecessary SELECT statements from prompt

2f6501a

Remove PostgreSQL schema from LLM prompt

cebd914

Display query failed error message in frontend

d70b1c6

Use labelIdentifier to load record display data

7ae8d06

Relation descriptions WiP

b01c9c4

Get display data using GraphQL WiP

c44fe4e

Create query using composite key definitions

f8f72e8

Simple display data GraphQL query works

ccb12f8

Remove recordDisplayDataById

8100e21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Text-to-SQL proof of concept #5788

Text-to-SQL proof of concept #5788

ad-elias commented Jun 9, 2024

github-actions bot commented Jun 9, 2024 •

edited

Loading

greptile-apps bot left a comment

FelixMalfait commented Jun 9, 2024

FelixMalfait left a comment

FelixMalfait Jun 9, 2024

greptile-apps bot left a comment

FelixMalfait Jun 16, 2024

FelixMalfait Jun 16, 2024

ad-elias Jun 21, 2024

FelixMalfait Jun 16, 2024

FelixMalfait left a comment

FelixMalfait Jun 23, 2024

FelixMalfait Jun 23, 2024

ad-elias Jun 24, 2024

FelixMalfait Jun 23, 2024

FelixMalfait Jun 23, 2024

		@@ -0,0 +1,95 @@
		import { Injectable } from '@nestjs/common';

		import { ChatOpenAI } from '@langchain/openai';

Text-to-SQL proof of concept #5788

Are you sure you want to change the base?

Text-to-SQL proof of concept #5788

Conversation

ad-elias commented Jun 9, 2024

github-actions bot commented Jun 9, 2024 • edited Loading

Welcome!

TODOs/FIXMEs:

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

FelixMalfait commented Jun 9, 2024

FelixMalfait left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FelixMalfait left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jun 9, 2024 •

edited

Loading