Generative AI Examples v0.9 Release Notes

OPEA Release Notes v0.9

What’s New in OPEA v0.9

• Broaden functionality

  • Provide telemetry functionalities for metrics and tracing using Prometheus, Grafana, and Jaeger
  • Initialize two Agent examples: AgentQnA and DocIndexRetriever
  • Support for authentication and authorization
  • Add Nginx Component to strengthen backend security
  • Provide Toxicity Detection Microservice
  • Support the experimental Fine-tuning microservice

• Enhancement

  • Align the Microservice format with the standards of OpenAI (Chat Completions, Fine-tuning... etc)
  • Enhance the performance benchmarking and evaluation for GenAI Examples, ex: TGI, resource allocation, ...etc
  • Enable support for launching container images as a non-root user
  • Use Llama-Guard-2-8B as default Guardrails model and bge-large-zh-v1.5 as default embedding model, mistral-7b-grok as default CodeTrans model
  • Add ProductivitySuite to provide access management and maintains user context

• Deployment

  • Support Red Hat OpenShift Container Platform (RHOCP)
  • GenAI Microservices Connector (GMC) successfully tested on Nvidia GPUs
  • Add Kubernetes support for AudioQnA and VisualQnA examples

• OPEA Docker Hub: https://hub.docker.com/u/opea

• GitHub IO: https://opea-project.github.io/latest/index.html

• Thanks for the external contribution from Sharan Shirodkar, Aishwarya Ramasethu
  , Michal Nicpon and Jacob Mansdorfer

Details

GenAIExamples

• ChatQnA

  • Update port in set_env.sh(040d2b7)
  • Fix minor issue in ChatQnA Gaudi docker README(a5ed223)
  • update chatqna dataprep-redis port(02a1536)
  • Add support for .md file in file upload in the chatqna-ui(7a67298)
  • Added the ChatQnA delete feature, and updated the corresponding README(09a3196)
  • fixed ISSUE-528(45cf553)
  • Fix vLLM and vLLM-on-Ray UT bug(cfcac3f)
  • set OLLAMA_MODEL env to docker container(c297155)
  • Update guardrail docker file path(06c4484)
  • remove ray serve(c71bc68)
  • Refine docker_compose for dataprep param settings(3913c7b)
  • fix chatqna guardrails(db2d2bd)
  • Support ChatQnA pipeline without rerank microservice(a54ffd2)
  • Update the number of microservice replicas for OPEA v0.9(e6b4fff)
  • Update set_env.sh(9657f7b)
  • add env for chatqna vllm(f78aa9e)

• Deployment

  • update manifests for v0.9(ba78b4c)
  • Update K8S manifest for ChatQnA/CodeGen/CodeTrans/DocSum(01c1b75)
  • Update benchmark manifest to fix errors(4fd3517)
  • Update env for manifest(4fa37e7)
  • update manifests for v0.9(08f57fa)
  • Add AudioQnA example via GMC(c86cf85)
  • add k8s support for audioqna(0a6bad0)
  • Update mainifest for FaqGen(80e3e2a)
  • Add kubernetes support for VisualQnA(4f7fc39)
  • Add dataprep microservice to chatQnA example and the e2e test(1c23d87)

• Documentation

  • [doc] Update README.md(c73e4e0)
  • doc fix: Update README.md to remove specific dicscription of paragraph-1(5a9c109)
  • doc: fix markdown in docker_image_list.md(9277fe6)
  • doc: fix markdown in Translation/README.md(d645305)
  • doc: fix markdown in SearchQnA/README.md(c461b60)
  • doc: fix FaqGen/README.md markdown(704ec92)
  • doc: fix markdown in DocSum/README.md(83712b9)
  • doc: fix markdown in CodeTrans/README.md(076bca3)
  • doc: fix CodeGen/README.md markdown(33f8329)
  • doc: fix markdown in ChatQnA/README.md(015a2b1)
  • doc: fix headings in markdown files(21fab71)
  • doc: missed an H1 in the middle of a doc(4259240)
  • doc: remove use of HTML for table in README(e81e0e5)
  • Update ChatQnA readme with OpenShift instructions(ed48371)
  • Convert HTML to markdown format.(14621f8)
  • Fix typo {your_ip} to {host_ip}(ad8ca88)
  • README fix typo(abc02e1)
  • fix script issues in MD file(acdd712)
  • Minor documentation improvements in the CodeGen README(17b9676)
  • Refine Main README(08eb269)
  • [Doc]Add a micro/mega service WorkFlow for DocSum(343d614)
  • Update README for k8s deployment(fbb81b6)

• Other examples

  • Clean deprecated VisualQnA code(87617e7)
  • Using TGI official release docker image for intel cpu(b2771ad)
  • Add VisualQnA UI(923cf69)
  • fix container name(5ac77f7)
  • Add VisualQnA docker for both Gaudi and Xeon using TGI serving(2390920)
  • Remove LangSmith from Examples(88eeb0d)
  • Modify the language variable to match language highlight.(f08d411)
  • Remove deprecated folder.(7dd9952)
  • update env for manifest(4fa37e7)
  • AgentQnA example(67df280)
  • fix tgi xeon tag(6674832)
  • Add new DocIndexRetriever example(566cf93)
  • Add env params for chatqna xeon test(5d3950)
  • ProductivitySuite Combo Application with REACT UI and Keycloak Authen(947cbe3)
  • change codegen tgi model(06cb308)
  • change searchqna prompt(acbaaf8)
  • minor fix mismatched hf token(ac324a9)
  • fix translation gaudi env(4f3be23)
  • Minor fixes for CodeGen Xeon and Gaudi Kubernetes codegen.yaml (c25063f)

• CI/CD/UT

  • update deploy_gmc logical in cd workflow(c016d82)
  • fix ghcr.io/huggingface/text-generation-inference tag(503a1a9)
  • Add GMC e2e in CD workflow(f45e4c6)
  • Fix CI test changed file detect issue([5...

Generative AI Examples v0.8 Release Notes

OPEA Release Notes v0.8

What’s New in OPEA v0.8

• Broaden functionality

  • Support frequently asked questions (FAQs) generation GenAI example
  • Expand the support of LLMs such as Llama3.1 and Qwen2 and support LVMs such as llava
  • Enable end-to-end performance and accuracy benchmarking
  • Support the experimental Agent microservice
  • Support LLM serving on Ray

• Multi-platform support

  • Release the Docker images of GenAI components under OPEA dockerhub and support the deployment with Docker
  • Support cloud-native deployment through Kubernetes manifests and GenAI Microservices Connector (GMC)
  • Enable the experimental authentication and authorization support using JWT tokens
  • Validate ChatQnA on multiple platforms such as Xeon, Gaudi, AIPC, Nvidia, and AWS

• OPEA Docker Hub: https://hub.docker.com/u/opea

Details

GenAIExamples

• ChatQnA

  • Add ChatQnA instructions for AIPC(26d4ff)
  • Adapt Vllm response format (034541)
  • Update tgi version(5f52a1)
  • Update README.md(f9312b)
  • Udpate ChatQnA docker compose for Dataprep Update(335362)
  • [Doc] Add valid micro-service details(e878dc)
  • Updates for running ChatQnA + Conversational UI on Gaudi(89ddec)
  • Fix win PC issues(ba6541)
  • [Doc]Add ChatQnA Flow Chart(97da49)
  • Add guardrails in the ChatQnA pipeline(955159)
  • Fix a minor bug for chatqna in docker-compose(b46ae8)
  • Support vLLM/vLLM-on-Ray/Ray Serve for ChatQnA(631d84)
  • Added ChatQnA example using Qdrant retriever(c74564)
  • Update TEI version v1.5 for better performance(f4b4ac)
  • Update ChatQnA upload feature(598484)
  • Add auto truncate for embedding and rerank(8b6094)

• Deployment

  • Add Kubernetes manifest files for deploying DocSum(831463)
  • Update Kubernetes manifest files for CodeGen(2f9397)
  • Add Kubernetes manifest files for deploying CodeTrans(c9548d)
  • Updated READMEs for kubernetes example pipelines(c37d9c)
  • Update all examples yaml files of GMC in GenAIExample(290a74)
  • Doc: fix minor issue in GMC doc(d99461)
  • README for installing 4 worklods using helm chart(6e797f)
  • Update Kubernetes manifest files for deploying ChatQnA(665c46)
  • Add new example of SearchQnA for GenAIExample(21b7d1)
  • Add new example of Translation for GenAIExample(d0b028)

• Other examples

  • Update reranking microservice dockerfile path (d7a5b7)
  • Update tgi-gaudi version(3505bd)
  • Refine README of Examples(f73267)
  • Update READMEs(8ad7f3)
  • [CodeGen] Add codegen flowchart(377dd2)
  • Update audioqna image name(615f0d)
  • Add auto-truncate to gaudi tei (8d4209)
  • Update visualQnA chinese version(497895)
  • Fix Typo for Translation Example(95c13d)
  • FAQGen Megaservice(8c4a25)
  • Code-gen-react-ui(1b48e5)
  • Added doc sum react-ui(edf0d1)

• CI/UT

  • Frontend failed with unknown timeout issue (7ebe78)
  • Adding Chatqna Benchmark Test(11a56e)
  • Expand tgi connect timeout(ee0dcb)
  • Optimize gmc manifest e2e tests(15fc6f)
  • Add docker compose yaml print for test(bb4230)
  • Refactor translation ci test (b7975e)
  • Refactor searchqna ci test(ecf333)
  • Translate UT for UI(284d85)
  • Enhancement the codetrans e2e test(450efc)
  • Allow gmc e2e workflow to get secrets(f45f50)
  • Add checkout ref in gmc e2e workflow(62ae64)
  • SearchQnA UT(268d58)

GenAIComps

• Cores

  • Support https for microservice(2d6772)
  • Enlarge megaservice request timeout for supporting high concurrency(876ca5)
  • Add dynamic DAG(f2995a)

• LLM

  • Optional vllm microservice container build(963755)
  • Refine vllm instruction(6e2c28)
  • Introduce 'entrypoint.sh' for some Containers(9ecc5c)
  • Support llamaindex for retrieval microservice and remove langchain(61795f)
  • Update tgi with text-generation-inference:2.1.0(f23694)
  • Fix requirements(f4b029)
  • Add vLLM on Ray microservice(ec3b2e)
  • Update code/readme/UT for Ray Serve and VLLM([dd939c](https://gith...

Generative AI Examples v0.7 Release Notes

OPEA Highlight

• Add 3 MegaService examples: Translation, SearchQnA and AudioQnA
• Add 4 MicroService and LLM supports llamaIndex, vllm, RayServe
• Enable Dataprep: extract info from table, image...etc
• Add HelmChart and GenAI Microservice Connector(GMC) test

GenAIExamples

• ChatQnA

  • ChatQnA supports Qwen2(422b4b)
  • Add no_proxy in docker compose yaml for micro services(99eb6a, 240587)
  • Fix DataPrep image build in ChatQnA(2fb070)
  • Add Nvidia GPU support for ChatQnA(e80e56)
  • Update ChatQnA docker_compose.yaml to fix downloads failing(e948a7, f2a943)
  • Chat QNA React UI with conversation history(b994bc)
  • Adapt Chinese characters(2f4723)

• Other examples

  • Refactor Translation Example(409c723)
  • Add AudioQnA with GenAIComps(b4d8e1)
  • Add SearchQnA with GenAIComps(6b76a9)
  • Add env for searchqna(d9b62a)
  • Supports ASR on HPU(2a4860)
  • Fix DocSum Gaudi building instructions(29de55)
  • Add image build job in docker compose e2e gaudi test in CI(4fecd4)

• CI

  • Add docker build job in manifest e2e workflow(c5f309)
  • Create reuse workflow for get-test-matrix in CI(961abb)
  • Enable new CI runner and improve manifest e2e test scripts(26d6ea)
  • Enable building latest megaservice image on push event in CI(a0b94b)
  • Fix the image build refer(01eed8)
  • Add build docker image option for test scripts(e32a51)
  • Add e2e test of chatqna(afcb3a), codetrans(295b818), codegen(960cf38), docsum(2e62ecc))

GenAIComps

• Cores

  • Add aio orchestrator to boost concurrent serving(db3b4f)
  • Add microservice level perf statistics(597b3c, ba1d11)
  • Add Gateway for Translation(1b654d)

• LLM

  • Support Qwen2 in LLM Microservice(3f5cde)
  • Fix the vLLM docker compose issues(3d134d)
  • Enable vLLM Gaudi support for LLM service based on officially habana vllm release(0dedc2)
  • Openvino support in vllm(7dbad0)
  • Support Ollama microservice(a00e36)
  • Support vLLM XFT LLM microservice(2a6a29, 309c2d, fe5f39)
  • Add e2e test for llm summarization tgi(e8ebd9)

• DataPrep

  • Support Dataprep(f7443f), embedding(f37ce2) microservice with Llama Index
  • Fix dataprep microservice path issue(e20acc)
  • Add milvus microservice(e85033)
  • Add Ray version for multi file process(40c1aa)
  • Fix dataprep timeout issue(61ead4)
  • Add e2e test for dataprep redis langchain(6b7bec)
  • Supported image summarization with LVM in dataprep microservice(86412c)
  • Enable conditional splitting for html files(e1dad1)
  • Added support for pyspark in dataprep microservice(a5eb14)
  • DataPrep extract info from table in the docs(953e78)
  • Added support for extracting info from image in the docs(e23745)

• Other Components

  • Add PGvector support in Vectorstores(1b7001) and Retriever(75eff6), Dataprep(9de3c7)
  • Add Mosec embedding(f76685) and reranking(a58ca4)
  • Add knowledge graph components(4c0afd)
  • Add LVMs LLaVA component(bd385b)
  • Add asr/tts components for xeon and hpu(cef6ea)
  • Add WebSearch Retriever Microservice(900178)
  • Add initial pii detection microservice(e38041)
  • Pinecone support for dataprep and retrieval microservice(8b6486)
  • Support prometheus metrics for opea microservices(758914), (900178)
  • Add no_proxy env for micro services(df0c11)
  • Enable RAGAS(8a670e)
  • Fix RAG performance issues(70c23d)
  • Support rerank and retrieval of RAG OPT(b51675)
  • Reranking using an optimized bi-encoder(574847)
  • Use parameter for retriever(358dbd), reranker(dfdd08)

• CI

  • CI optimization to support multiple test for single kind of service(38f646)
  • Update CI to support dataprep_redis path level change(5c0773)
  • Enable python coverage(cd91cf)
  • Add codecov(da2689)
  • Enable microservice docker images auto build and push(16c5fd)

GenAIEvals

• Enable autorag to automatically generate the evaluation dataset and evaluate the RAG system(b24bff)
• Support document summar...

Generative AI Examples v0.6 Release Notes

OPEA Highlights

• Add 4 MegaService examples: CodeGen, ChatQnA, CodeTrans and Docsum, you can deploy them on Kubernetes
• Enable 10 microservices for LLM, RAG, security...etc
• Support text generation, code generation and end-to-end evaluation

GenAIExamples

• Build 4 reference solutions for some classic GenAI applications, like code generation, chat Q&A, code translation and document summarization, through orchestration interface in GenAIComps.
• Support seamlessly deployment on Intel Xeon and Gaudi platform through Kubernetes and Docker Compose.

GenAIComps

• Activate a suite of microservices including ASR, LLMS, Rerank, Embedding, Guardrails, TTS, Telemetry, DataPrep, Retrieval, and VectorDB. ASR functionality is fully operational on Xeon architecture, pending readiness on Gaudi. Retrieval capabilities are functional on LangChain, awaiting readiness on LlamaIndex. VectorDB functionality is supported on Redis, Chroma, and Qdrant, with readiness pending on SVS.
• Added 14 file formats support in data preparation microservices and enabled Safeguard of conversation in guardrails.
• Added the Ray Gaudi Supported for LLM Service.

GenAIEvals

• Add evaluating the models on text-generation tasks(lm-evaluation-harness) and coding tasks (bigcode-evaluation-harness)
• Add end-to-end evaluation with microservice

GenAIInfra

• Add Helm Charts redis-vector-db, TEI, TGI and CodeGen for deploying GenAIExamples on Kubernetes
• Add Manifests for deploying GenAIExamples CodeGen, ChatQnA and Docsum on Kubernetes and on Docker Compose

Generative AI Examples v0.1 Release Notes

• Highlights
• Examples

Highlights

• Provides a collective list of Generative AI (GenAI) and Retrieval-Augmented Generation (RAG) examples such as chatbot with question and answering (ChatQnA), code generation (CodeGen), document summary (DocSum), etc.
• All the examples are well-validated and optimized on Intel platforms.
• Use ecosystem-compliant APIs to build the end-to-end GenAI examples.
• Deploy the GenAI examples with performance on Intel platforms.

Examples

• ChatQnA: an example of chatbot for question and answering through retrieval argumented generation (RAG)
• CodeGen: an example of copilot designed for code generation in Visual Studio Code.
• DocSum: an example of chatbot for summarizing the content of documents or reports.
• SearchQnA: an example of chatbot for using search engine to enhance QA quality.
• VisualQnA is an example of chatbot for question and answering based on the images.

Validated Configurations

• Python 3.8, 3.9, 3.10, 3.11
• Ubuntu 20.04
• PyTorch 2.2.0+cpu 2.1.0+cpu
• Intel® Extension for PyTorch 2.2.0+cpu, 2.1.0+cpu
• TGI Gaudi 1.2.1
• LangChain 0.1.12