[HUDI-8769] Limit instantsToArchive to avoid expensive locks blocking ingestion and table services

[vinishjail97]

Change Logs

Archival takes a lock before archiving instants when a large timeline with tens of thousands instants is pruned to few hundreds this blocks other table services from completion because they can't acquire the lock.

Impact

Avoids blocking ingestion and other table services for archiving large timelines.

Risk level (write none, low medium or high below)

Low

Documentation Update

  public static final ConfigProperty<Long> ARCHIVE_LIMIT_INSTANTS = ConfigProperty
      .key("hoodie.archive.limit.instants")
      .defaultValue(Long.MAX_VALUE)
      .markAdvanced()
      .sinceVersion("0.16.0")
      .withDocumentation("If enabled, archival will limit the instants upto this value."
          + "This is useful when a large timeline containing tens of thousands instants that need to archived");

Contributor's checklist

• Read through contributor's guide
• Change Logs and Impact were stated clearly
• Adequate tests were added if applicable
• CI passed

[vinishjail97]

Where do I add tests for v1 archival ? They seem to have been deleted in 1.0 and moved to a new location.

[nsivabalan]

this might contradict with "hoodie.keep.min.commits". So, the question we might need to ask ourselves is, why not adjusting "hoodie.keep.min.commits" and "hoodie.keep.max.commits" help here?

[nsivabalan]

can we add tests please

[danny0405]

this might contradict with "hoodie.keep.min.commits". So, the question we might need to ask ourselves is, why not adjusting "hoodie.keep.min.commits" and "hoodie.keep.max.commits" help here?

I'm wondering why there are so many active instants on the timeline, the ingestion still not perform well if there are too many small files within the timeline. And listing cost should be huge.

[vinishjail97]

this might contradict with "hoodie.keep.min.commits". So, the question we might need to ask ourselves is, why not adjusting "hoodie.keep.min.commits" and "hoodie.keep.max.commits" help here?

I'm wondering why there are so many active instants on the timeline, the ingestion still not perform well if there are too many small files within the timeline. And listing cost should be huge.

The table can be in a bad state and user enabled table services with the right configs.

[hudi-bot]

CI report:

• 0346215 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[danny0405]

this might contradict with "hoodie.keep.min.commits". So, the question we might need to ask ourselves is, why not adjusting "hoodie.keep.min.commits" and "hoodie.keep.max.commits" help here?

I'm wondering why there are so many active instants on the timeline, the ingestion still not perform well if there are too many small files within the timeline. And listing cost should be huge.

The table can be in a bad state and user enabled table services with the right configs.

Maybe a better solution is just archive the instants in one shot from the HUDI CLI cmd.