Skip to content

[AMORO-3775] Add support for metric-based refresh event trigger in TableRuntimeRefreshExecutor #3776

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[AMORO-3775] Add support for metric-based refresh event trigger in TableRuntimeRefreshExecutor #3776

wants to merge 2 commits into from

Conversation

@Jzjsnow
Copy link
Contributor

@Jzjsnow Jzjsnow commented Sep 9, 2025 •
edited
Loading

Why are the changes needed?

Close #3775.

Brief change log

Add support for MSE based refresh event:

  • Support for calculating partition filesize mean square error based on the loaded metadata.
  • Filter partitions need to be optimized based on threshold and trigger pendingInput evaluation if necessary.

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@Jzjsnow Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch 2 times, most recently from cd57764 to c8734fb Compare September 29, 2025 09:11
@Jzjsnow Jzjsnow changed the title [AMORO-3775] Add support for pluggable refresh event in TableRuntimeRefreshExecutor [AMORO-3775] Add support for metric-based refresh event trigger in TableRuntimeRefreshExecutor Sep 29, 2025
@Jzjsnow Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch from c8734fb to c352653 Compare September 29, 2025 09:32
@Jzjsnow Jzjsnow force-pushed the add_support_for_pluggable_refresh_event branch from c352653 to f00825b Compare September 29, 2025 09:41
@xxubai
Copy link
Contributor

xxubai commented Oct 27, 2025

Can we move forward with this feature now? @Jzjsnow @klion26

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

module:ams-server Ams server module module:common

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Subtask]: Add support for Metadata Metric-Driven refresh event

2 participants

@Jzjsnow @xxubai