[Feature](multimodal_embed) Support multimodal(file) embed by linrrzqqq · Pull Request #62147 · apache/doris

linrrzqqq · 2026-04-07T03:33:00Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

Support image/video/audio embedding from json.

Supported multimodal_embed AI provider : Gemini(image, video, audio), Qwen(image, video), Voyage(image, video), Jina(image)

Supported authentication methods include IAM + EXTERNAL_ID / AK + SK

-- ima role, video
mysql> SELECT ARRAY_SIZE(
    ->     EMBED(
    ->     'qwen_mul_embed',
    ->     CAST('{
    '>         "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/video/45944deac7d96c872f559f2ef94ea0a9.mp4",
    '>             "content_type": "video/mp4",
    '>             "provider": "AWS",
    '>             "endpoint": "s3.us-east-1.amazonaws.com",
    '>             "region": "us-east-1",
    '>             "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test",
    '>             "external_id": "1001"
    '>         }' AS JSON)
    ->     )
    -> ) AS video_embed_size;
+------------------+
| video_embed_size |
+------------------+
|             2560 |
+------------------+

-- ima role, video 
mysql> SELECT ARRAY_SIZE(
    ->     EMBED(
    ->         'qwen_mul_embed',
    ->         CAST('{
    '>             "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/image/ai-agg.png",
    '>             "content_type": "image/png",
    '>             "provider": "AWS",
    '>             "endpoint": "s3.us-east-1.amazonaws.com",
    '>             "region": "us-east-1",
    '>             "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test",
    '>              "external_id": "1001"
    '>         }' AS JSON)
    ->     )
    -> ) AS img_embed_size;
+----------------+
| img_embed_size |
+----------------+
|           2560 |
+----------------+

-- gemini multimodal embed
mysql> SELECT ARRAY_SIZE(
    ->     EMBED(
    ->         'gemini_mul_embed',
    ->         CAST('{
    '>             "uri": "s3://selectdb-qa-test-3/lzq-multimodal-test/image/ai-agg.png",
    '>             "content_type": "image/png",
    '>             "provider": "AWS",
    '>             "endpoint": "s3.us-east-1.amazonaws.com",
    '>             "region": "us-east-1",
    '>             "role_arn": "arn:aws:iam::447051187841:role/lzq-role-test",
    '>              "external_id": "1001"
    '>         }' AS JSON)
    ->     )
    -> ) AS img_embed_size;
+----------------+
| img_embed_size |
+----------------+
|           3072 |
+----------------+

-- ak, sk
Doris> SELECT ARRAY_SIZE(
    ->     EMBED(
    ->         'qwen_mul_embed',
    ->         CAST('{
    '>             "uri": "s3://<bucket>/<key>/test_img.png",
    '>             "content_type": "image/png",
    '>             "endpoint": "cos.ap-hongkong.myqcloud.com",
    '>             "region": "ap-hongkong",
    '>             "ak": "<ak>",
    '>             "sk": "<sk>"
    '>         }' AS JSON)
    ->     )
    -> ) AS img_embed_size_ak_sk;
+----------------------+
| img_embed_size_ak_sk |
+----------------------+
|                 2560 |
+----------------------+

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-04-07T03:33:05Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

...e-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ai/AIFunction.java

linrrzqqq · 2026-04-07T07:36:24Z

run buildall

hello-stephen · 2026-04-07T08:14:32Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.48% (1798/2291)
Line Coverage	64.14% (32271/50310)
Region Coverage	64.99% (16174/24886)
Branch Coverage	55.45% (8618/15542)

linrrzqqq · 2026-04-07T12:36:27Z

run buildall

hello-stephen · 2026-04-07T13:10:43Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.48% (1798/2291)
Line Coverage	64.15% (32275/50310)
Region Coverage	65.04% (16185/24886)
Branch Coverage	55.48% (8623/15542)

linrrzqqq · 2026-04-08T02:06:29Z

run buildall

hello-stephen · 2026-04-08T03:31:12Z

FE UT Coverage Report

Increment line coverage 17.74% (11/62) 🎉
Increment coverage report
Complete coverage report

linrrzqqq · 2026-04-08T06:35:21Z

run beut

linrrzqqq · 2026-04-08T09:15:51Z

run buildall

hello-stephen · 2026-04-08T09:58:27Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.48% (1798/2291)
Line Coverage	64.13% (32265/50312)
Region Coverage	65.09% (16242/24954)
Branch Coverage	55.57% (8675/15610)

hello-stephen · 2026-04-08T10:28:52Z

FE UT Coverage Report

Increment line coverage 17.74% (11/62) 🎉
Increment coverage report
Complete coverage report

linrrzqqq · 2026-04-09T03:52:43Z

run beut

linrrzqqq · 2026-04-09T07:42:28Z

run buildall

hello-stephen · 2026-04-09T09:32:06Z

FE UT Coverage Report

Increment line coverage 17.74% (11/62) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-09T09:35:46Z

BE UT Coverage Report

Increment line coverage 52.17% (240/460) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.99% (20133/37996)
Line Coverage	36.55% (189373/518053)
Region Coverage	32.79% (146907/448032)
Branch Coverage	33.90% (64301/189666)

hello-stephen · 2026-04-09T12:25:11Z

BE Regression && UT Coverage Report

Increment line coverage 42.57% (189/444) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.63% (27398/37211)
Line Coverage	57.28% (295810/516471)
Region Coverage	54.39% (245919/452165)
Branch Coverage	56.12% (106765/190248)

HappenLee

LGTM

github-actions · 2026-04-10T07:10:59Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-04-10T07:11:02Z

PR approved by anyone and no changes requested.

linrrzqqq · 2026-04-12T19:21:07Z

run buildall

hello-stephen · 2026-04-12T20:11:01Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	78.48% (1798/2291)
Line Coverage	64.07% (32258/50345)
Region Coverage	65.05% (16241/24967)
Branch Coverage	55.58% (8681/15620)

linrrzqqq · 2026-04-13T04:18:11Z

run beut

hello-stephen · 2026-04-13T08:22:40Z

BE Regression && UT Coverage Report

Increment line coverage 75.90% (337/444) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	71.92% (26772/37224)
Line Coverage	54.86% (283427/516648)
Region Coverage	51.73% (233990/452310)
Branch Coverage	53.22% (101276/190296)

linrrzqqq · 2026-04-13T08:36:51Z

run buildall

hello-stephen · 2026-04-13T09:57:14Z

FE UT Coverage Report

Increment line coverage 17.46% (11/63) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-04-13T11:59:53Z

BE Regression && UT Coverage Report

Increment line coverage 75.90% (337/444) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.65% (27409/37214)
Line Coverage	57.32% (296074/516527)
Region Coverage	54.50% (246447/452217)
Branch Coverage	56.18% (106882/190233)

morrySnow reviewed Apr 7, 2026

View reviewed changes

...e-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/ai/AIFunction.java Outdated Show resolved Hide resolved

linrrzqqq force-pushed the embed-backup branch from fb144d9 to d1d4f58 Compare April 7, 2026 03:37

linrrzqqq force-pushed the embed-backup branch from 466a50c to 5ef38a0 Compare April 8, 2026 01:47

linrrzqqq force-pushed the embed-backup branch from 14fb2fd to 3cf8ecd Compare April 9, 2026 03:52

HappenLee previously approved these changes Apr 10, 2026

View reviewed changes

Gabriel39 previously approved these changes Apr 10, 2026

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 10, 2026

github-actions bot added the reviewed label Apr 10, 2026

linrrzqqq added 4 commits April 13, 2026 01:01

[embed-test]

cca248a

aksk

8ab0d14

move rigsterUesdAIResource to Resource.java

cface1c

fix p0

0abec88

linrrzqqq dismissed stale reviews from Gabriel39 and HappenLee via fd7cffd April 12, 2026 18:54

linrrzqqq force-pushed the embed-backup branch from 3cf8ecd to fd7cffd Compare April 12, 2026 18:54

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 12, 2026

linrrzqqq force-pushed the embed-backup branch from fd7cffd to 21aa58a Compare April 12, 2026 19:03

add beut

1aed12f

linrrzqqq force-pushed the embed-backup branch from 21aa58a to 1aed12f Compare April 13, 2026 08:34

Conversation

linrrzqqq commented Apr 7, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Apr 7, 2026

Uh oh!

Uh oh!

linrrzqqq commented Apr 7, 2026

Uh oh!

hello-stephen commented Apr 7, 2026

Cloud UT Coverage Report

Uh oh!

linrrzqqq commented Apr 7, 2026

Uh oh!

hello-stephen commented Apr 7, 2026

Cloud UT Coverage Report

Uh oh!

linrrzqqq commented Apr 8, 2026

Uh oh!

hello-stephen commented Apr 8, 2026

FE UT Coverage Report

Uh oh!

linrrzqqq commented Apr 8, 2026

Uh oh!

linrrzqqq commented Apr 8, 2026

Uh oh!

hello-stephen commented Apr 8, 2026

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Apr 8, 2026

FE UT Coverage Report

Uh oh!

linrrzqqq commented Apr 9, 2026

Uh oh!

linrrzqqq commented Apr 9, 2026

Uh oh!

hello-stephen commented Apr 9, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Apr 9, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Apr 9, 2026

BE Regression && UT Coverage Report

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 10, 2026

Uh oh!

github-actions bot commented Apr 10, 2026

Uh oh!

linrrzqqq commented Apr 12, 2026

Uh oh!

hello-stephen commented Apr 12, 2026

Cloud UT Coverage Report

Uh oh!

linrrzqqq commented Apr 13, 2026

Uh oh!

hello-stephen commented Apr 13, 2026

BE Regression && UT Coverage Report

Uh oh!

linrrzqqq commented Apr 13, 2026

Uh oh!

hello-stephen commented Apr 13, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Apr 13, 2026

BE Regression && UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!