Add schema to table location #134

kmcclenn · 2024-07-11T00:41:48Z

Summary

Issue Changed table locations to include the scheme.

Changes

Changed both the implementation of creating the table location (and dealing with its ramifications) and the tests that test the table location. Allows all table locations to include the schema (eg. hdfs://tmp/... instead of tmp/...). Furthermore, added this fix for the old constructTablePath method and the new allocateTableStorage method. Removed outdated overriding methods for Local and HDFS storage that don't include the scheme.

Testing Done

Manually Tested on local docker setup. Please include commands ran, and their output.
Added new tests for the changes made.
Updated existing tests to reflect the changes made.
No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
Some other form of testing like staging or soak time in production. Please explain.

Changed tests to reflect the table locations now containing the schema.

Additional Information

Breaking Changes
Deprecations
Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

HotSushi

Thanks for the PR @kmcclenn I'm curious about the necessity of this change. There are some prereqs before we tackle #121 .

HotSushi · 2024-07-11T16:37:12Z

apps/spark/src/main/java/com/linkedin/openhouse/jobs/spark/Operations.java

@@ -235,7 +235,8 @@ public void runRetention(
  }

  private Path getTrashPath(String path, String filePath, String trashDir) {
-    return new Path(filePath.replace(path, new Path(path, trashDir).toString()));
+    return new Path(
+        filePath.replace(new Path(path).toString(), new Path(path, trashDir).toString()));


why did we need this change?

With the schema, the string path was file:///tmp/.... However, we were trying to replace it with something like file:/tmp/..., because the Path type only allows for single forward slashes. So, I converted it to a Path to standardize the format to file:/tmp/... so they would match.

HotSushi · 2024-07-11T16:39:42Z

apps/spark/src/test/java/com/linkedin/openhouse/jobs/spark/OperationsTest.java

-      log.info("Detected {} orphan files", orphanFiles.size());
-      for (String of : orphanFiles) {
+      List<String> orphanFilesWithoutMetadata =
+          orphanFiles.stream()


why metadata.json files filtered out?

Seems that the tests were meant for locations without schema: See this slack message from Stas

HotSushi · 2024-07-11T16:40:56Z

cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/BaseStorage.java

@@ -60,10 +60,6 @@ public String allocateTableLocation(
      String databaseId, String tableId, String tableUUID, String tableCreator) {
    Preconditions.checkArgument(databaseId != null, "Database ID cannot be null");
    Preconditions.checkArgument(tableId != null, "Table ID cannot be null");
-    Preconditions.checkArgument(tableUUID != null, "Table UUID cannot be null");


this change seem regressive, its valid to ensure tableUUID is not null/ storageproperties is correctly configured

Yes, I was a bit confused here. I removed it because it was failing this test because the STAGED_TABLE_DTO did not have a UUID; there is probably a better solution here but I wasn't sure what was best

HotSushi · 2024-07-11T16:42:59Z

cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/hdfs/HdfsStorage.java

-   * @return the table location
-   */
-  @Override
-  public String allocateTableLocation(


we ought to be careful with this. The reason we didn't make this change earlier was to ensure backward compatibilty. I would hold off checking this in until we keep the behavior compatible in li-openhouse. Could you make a change there to not call super.allocateTableStorage?

HotSushi · 2024-07-11T16:44:08Z

cluster/storage/src/main/java/com/linkedin/openhouse/cluster/storage/local/LocalStorage.java

-   * href="https://github.com/linkedin/openhouse/issues/121">
-   */
-  @Override
-  public String allocateTableLocation(


we have to be careful with this change as well. tables-test-fixtures uses non-schemed paths. Please validate tables-test-fixtures continue to work after this change cc: @jiang95-dev

HotSushi · 2024-07-11T16:49:44Z

...les/src/main/java/com/linkedin/openhouse/tables/repository/impl/InternalRepositoryUtils.java

-        storageManager.getDefaultStorage().getClient().getRootPrefix(),
-        databaseID,
-        tableId + "-" + tableUUID);
+    String endpoint = storageManager.getDefaultStorage().getClient().getEndpoint();


There's no need to make this change yet. constructTablePath is only getting used in CTAS (and only for hdfs). non-hdfs storages need much more things to correctly support ctas: BDP-23723

HotSushi · 2024-07-11T16:52:50Z

services/tables/src/main/java/com/linkedin/openhouse/tables/utils/TableUUIDGenerator.java

@@ -144,8 +145,10 @@ private void validatePathOfProvidedRequest(
        extractFromTblPropsIfExists(databaseId + "." + tableId, tableProperties, TBL_RAW_KEY);

    java.nio.file.Path previousPath =
-        InternalRepositoryUtils.constructTablePath(
-            storageManager, dbIdFromProps, tblIdFromProps, tableUUIDProperty);
+        Paths.get(


lets avoid making change to this class. CTAS support is not present for non-hdfs storages, and this needs to be solved thoughtfully: BDP-23723

see doesPathExist(..) UnsupportedOperartionException

kmcclenn · 2024-07-11T17:27:15Z

@HotSushi Thanks for the comments. I originally started the PR because the ADLS storage was failing without schema in the table location. I added changes for constructTablePath and all of the related tests before I realized that this was changed in main by the allocateTableLocation function. So it may not be needed anymore, especially if I will be breaking the backwards compatibility with the PR. Do you think I should just close it and move forward with allocateTableLocation for ADLS?

HotSushi · 2024-07-11T19:31:43Z

@HotSushi Thanks for the comments. I originally started the PR because the ADLS storage was failing without schema in the table location. I added changes for constructTablePath and all of the related tests before I realized that this was changed in main by the allocateTableLocation function. So it may not be needed anymore, especially if I will be breaking the backwards compatibility with the PR. Do you think I should just close it and move forward with allocateTableLocation for ADLS?

yes allocateTableLocation is the way to go about it. We can pursue this PR later once backward compatibility is ensured.

jiang95-dev · 2024-07-12T16:59:12Z

Thanks @kmcclenn. You can close this pr if allocateTableLocation solves the problem.
Thanks @HotSushi. I have some questions about the design and plan:

Are we going to remove allocateTableLocation once the backward compatibility is ensured? I feel it is still needed to parse the location whether or not it is with scheme?
What will be the state for the backward compatibility? Old tables storing non-schemed tableLocation, and new tables storing schemed tableLocation, and OH server can read/write both correctly?

kmcclenn added 4 commits July 10, 2024 14:41

add schema to table location

6ae187f

fixed broken tests for delete orphans

dfe7fba

passing all tests

b559c45

fixed issues related to allocateTableLocation

b0da6cd

kmcclenn changed the title ~~Kai/add schema to table location~~ Add schema to table location Jul 11, 2024

kmcclenn closed this Jul 11, 2024

kmcclenn reopened this Jul 11, 2024

HotSushi requested changes Jul 11, 2024

View reviewed changes

sumedhsakdeo requested review from teamurko and autumnust July 12, 2024 04:49

kmcclenn closed this Jul 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add schema to table location #134

Add schema to table location #134

kmcclenn commented Jul 11, 2024

HotSushi left a comment

HotSushi Jul 11, 2024

kmcclenn Jul 11, 2024

HotSushi Jul 11, 2024

kmcclenn Jul 11, 2024

HotSushi Jul 11, 2024

kmcclenn Jul 11, 2024

HotSushi Jul 11, 2024

HotSushi Jul 11, 2024

HotSushi Jul 11, 2024

HotSushi Jul 11, 2024

kmcclenn commented Jul 11, 2024

HotSushi commented Jul 11, 2024

jiang95-dev commented Jul 12, 2024

Add schema to table location #134

Add schema to table location #134

Conversation

kmcclenn commented Jul 11, 2024

Summary

Changes

Testing Done

Additional Information

HotSushi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmcclenn commented Jul 11, 2024

HotSushi commented Jul 11, 2024

jiang95-dev commented Jul 12, 2024