refactor: hug llamaindex (#91)

* use binary as id

* add llamaindex

* add document store

* update

* update

* refine flow

* feat: add tidb vector store

* fix: align table schema to llamaindex node definition

* refactor: align document store to llamaindex dir

* llamdaindex pipeline example

* update

* fix

* add index store

* fix: refine TiDBVectorStore

* fix: should process with uuid_to_bin when insert

* refactor: update schema

* fix sql

* update

* implement importing and indexing tasks logic

* revert storage extension

* update pnpm lock

* fix PdfLoader type

* update

* wrapper API error

* refine document_task schedule API and errors handle

* update

* config: update initial config

* fix: fix get index bug

* update

* update

* update

* update

* fix

* using BLOB_READ_WRITE_TOKEN first if it exists

* fix build ci

* exclude pdfjs canvas dep

* try to fix exceeded the unzipped maximum size of 250 MB

* fix

* fix

* update

* remove splitter abstraction

* remove prompting abstraction

* fix status check

* todo: fix list source API

* fix user not found error message

* refine document import task error handle

* update

* fix ui

* fix ui

* fix

* fix

* fix

* update cronjob

* update cronjob maxDuration

* update speed

* fix

* fix

* update

* update index process duration limit

* fix

* fix

* fix log

* fix sql

* update

* fix retrieve

* update docs

* fix

* fix

* update prompt

* fix context retrieving

* remove log

* speed up!

* fix error messages length

* fill document names

* fix old api

* add url_key to chat

* fix delete chat

* feat: add llm reranker (#96)

* feat: add llm reranker

* support provide query by query string

* fix typo

* feat: support select chat engine

* fix

* reorder fields

* fix source title

* feat: add bitdeer llm (#97)

* feat: add bitdeer llm

* fix: refine error handle

* add index config page

* fix dashboard page

* fix

* fix

* feat: refine ann index preload (#99)

* feat: refine ann index preload

* fix: remove old tables

* remove the old status code

* refactor authentication provider

* migrate settings

* migrate extensions

* add dockerfile

* add source operations

* refactor: refine v1 APIs (#101)

* refactor: refine v1 APIs

* retry required admin

* fix ci

* protect settings API

* feat: impl readonly index config page

* fix: chat session id

* fix: add public dir in dockerfile

* refactor: remove v1 dir in core (#102)

* remove v1 dir

* merge

* chore: move operations to client dir

* chore: remove docs

* chore: remove unused

* feat: add index summary stats

* fix initial table schema (#103)

* fix ci

* fix settings api

* update index editor pages

* fix settings page refresh

* feat: enable llm provider as default reranker

* fix ci

* refactor: use services replacing jobs (#104)

* move import processor to DocumentImportService

* move index processor to DocumentIndexService

* refine chats API

* add github action to build docker image (#106)

* refactor: optimize settings

* fix: flatten empty array settings

* fix: upload image with local fs storage

* try to fix Dockerfile

* try to fix Dockerfile

* prevent re-fetch chats

* ui: update side nav

* ui: support enable index

* fix settings

* update github action

* support update chat engine

* try fix chat stream

* fix: settings schema

* ui: remove hard coded tidb.ai

* ui: support update chat engine's llm and reranker

* feat: support bitdeer embedding and chat api (#107)

* feat: support bitdeer embedding and chat api

* test

* ui: update form default values

---------

Co-authored-by: Jagger <[email protected]>
Co-authored-by: WD <[email protected]>

.dockerignore

@@ -0,0 +1,3 @@
+node_modules
+.next
+.store

.github/workflows/build-image.yml

@@ -0,0 +1,34 @@
+name: Build and Push Docker Image
+
+on:
+    push:
+      branches:
+        - main
+    workflow_dispatch:
+
+jobs:
+    build:
+      runs-on: ubuntu-latest
+      steps:
+        -
+          name: Checkout
+          uses: actions/checkout@v4
+        -
+          name: Login to Docker Hub
+          uses: docker/login-action@v3
+          with:
+            username: ${{ secrets.DOCKERHUB_USERNAME }}
+            password: ${{ secrets.DOCKERHUB_TOKEN }}
+        -
+          name: Set up Docker Buildx
+          uses: docker/setup-buildx-action@v3
+        -
+          name: Build and push
+          uses: docker/build-push-action@v5
+          with:
+            context: .
+            file: ./Dockerfile
+            push: true
+            tags: |
+                ${{ secrets.DOCKERHUB_USERNAME }}/tidb.ai:latest
+                ${{ secrets.DOCKERHUB_USERNAME }}/tidb.ai:${{ github.sha }}

Dockerfile

@@ -0,0 +1,36 @@
+FROM node:20-alpine AS base
+# 1. Install dependencies only when needed
+FROM base AS deps
+# Check https://github.com/nodejs/docker-node/tree/b4117f9333da4138b03a546ec926ef50a31506c3#nodealpine to understand why libc6-compat might be needed.
+RUN apk add --no-cache libc6-compat
+WORKDIR /app
+# Install dependencies based on the preferred package manager
+COPY . .
+RUN corepack enable
+RUN pnpm i
+
+# 2. Rebuild the source code only when needed
+FROM base AS builder
+WORKDIR /app
+COPY --from=deps /app/node_modules ./node_modules
+COPY . .
+# This will do the trick, use the corresponding env file for each environment.
+# COPY .env.production.sample .env.production
+RUN corepack enable
+RUN STANDALONE_BUILD=1 pnpm run build
+
+# 3. Production image, copy all the files and run next
+FROM base AS runner
+WORKDIR /app
+ENV NODE_ENV=production
+RUN corepack enable
+RUN addgroup -g 1001 -S nodejs
+RUN adduser -S nextjs -u 1001
+# Automatically leverage output traces to reduce image size
+# https://nextjs.org/docs/advanced-features/output-file-tracing
+COPY --from=builder --chown=nextjs:nodejs /app/.next/standalone ./
+COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
+COPY public ./public
+USER nextjs
+EXPOSE 3000
+CMD ["node", "server.js"]

README.md

@@ -66,6 +66,6 @@ For deploying the application to production, there are many options available:
 * Adapt the system for AWS Redrock Claude3, with an anticipated completion date of April 15th.
 * Finalize the "How It Works" series of documentation by April 30th.
 
-
 ## License
+
 TiDB.AI is open-source under the Apache License, Version 2.0. You can [find it here](/LICENSE.txt).