[SUPPORT] Wrong file path for commit-marker-file in Consistent-Bucket-Index layout

[TheR1sing3un]

In Consistent-Bucket-Index file layout:
there are two type of file for each versioned bucket metadata:

1. {instant}.hashing_metadata: created in each bucket resizing.
2. {instant}.commit: created after instant is completed.

All these two type of files should all be stored in this path:

When loadMetadata, will load all these two type of files from above path.

But, I find that the {instant}.commit file will not be stored in this hash-metadata path.

This file will be stored in data path, which is {basePath}/{partitionPath}/{instant}.commit. So when ConsistentBucketIndexUtils load metadata for a partition, all files of this type will not be loaded because they are stored in the wrong location.

For example:

.hashing_metadata

.commit

Tips before filing an issue

• Have you gone through our FAQs?

• Join the mailing list to engage in conversations and get faster support at [email protected].

• If you have triaged this as a bug, then file an issue directly.

Describe the problem you faced

A clear and concise description of the problem.

To Reproduce

Steps to reproduce the behavior:

Expected behavior

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version :

• Spark version :

• Hive version :

• Hadoop version :

• Storage (HDFS/S3/GCS..) :

• Running on Docker? (yes/no) :

Additional context

Add any other context about the problem here.

Stacktrace

Add the stacktrace of the error.