Fixed floorDiv(long,int) to floorDiv(long,long) for Java8 compatibility.

Author: dstreev

Writerside/topics/Warehouse-Plans.md

@@ -1,7 +1,6 @@
 # Warehouse Plans
 
-Warehouse Plans are the way you can control how 'each' Database is translated between clusters or storage environments.
-Warehouse plans allow you to control where the data will be translated to.
+Warehouse Plans in "hms-mirror" are a database-level configuration mechanism designed to manage and map storage locations within a Hive database during metadata migration. They involve reviewing the database metadata to identify all storage locations (e.g., table and partition locations for both External and Managed tables) and mapping these to predefined Warehouse Plan locations. The intersection of this metadata and the Warehouse Plan is then used to construct **Global Location Maps**, which "hms-mirror" uses internally to provide a consistent mapping between storage systems across clusters or within a single cluster.
 
 There are three types of 'Warehouse Plans'.
 
@@ -30,3 +29,113 @@ When you choose to use non-standard locations for 'partition specs', we can't bu
 we will throw an 'error' for the offending table and describe to imbalance.  You can either fix/adjust the dataset OR choose
 to use the `SQL` data movement strategy.
 
+## Warehouse Plans Explained
+
+### What Are Warehouse Plans For?
+
+- **Purpose**: At the database level, Warehouse Plans define how storage locations in the source database metadata are translated to target locations. This process ensures that the migrated metadata aligns with the desired storage structure on the target system. Unlike the earlier [`globalLocationMap`](Release-Notes.md#global-location-maps), which offered a simpler, global key-value mapping, Warehouse Plans operate specifically at the database scope, providing a more targeted and detailed approach.
+- [**Database Warehouse Plans**](Database-Warehouse-Plans.md) : These are the core of the feature, specifying location mappings for a single database. The documentation implies that ["Global Warehouse Plans"](Global-Warehouse-Plans.md) may extend this concept across multiple databases, but the primary focus is on individual database-level planning.
+
+The resulting Global Location Maps, derived from Warehouse Plans, serve as an internal framework for "hms-mirror" to handle location translations during migration, ensuring consistency between source and target storage systems.
+
+### What to Use Warehouse Plans For?
+
+Warehouse Plans are primarily designed for scenarios where storage locations within a database need to be systematically mapped or reorganized during a migration. Here’s an explanation of their use cases, reflecting their database-level scope and original intent for `STORAGE_MIGRATION`, as well as their broader applicability:
+
+1. **Primary Use Case: Storage Migration Within a Cluster (STORAGE_MIGRATION)**:
+    - **Scenario**: You’re migrating the storage layer behind a database’s metadata from HDFS to Ozone (or another system like an encrypted zone), as highlighted in the [`STORAGE_MIGRATION` strategy](storage_migration.md).
+    - **Use**: Define a Warehouse Plan for the database to map all existing locations (e.g., `/apps/hive/warehouse/my_db`) to a new target (e.g., `ofs://ozone1/vol1/bucket1/my_db`). The metadata is reviewed, and all table and partition locations are updated accordingly. The resulting Global Location Maps ensure the migration reflects the new storage system.
+    - **Example**: For a database `finance_db`:
+      ```yaml
+      databaseWarehousePlans:
+        finance_db:
+          EXTERNAL_TABLE: "/finance_db/ext"
+          MANAGED_TABLE: "/finance_db/managed"
+      ```
+      This maps all External and Managed table locations within `finance_db` to the specified Ozone paths.
+    - The Ozone Namespace is pulled from the `targetNamespace` which can be set in the config or via the Web UI.
+      ```yaml
+      transfer:
+        targetNamespace: ofs://myOzone
+      ```
+
+2. **Reorganizing Storage During Schema Migration (SCHEMA_ONLY)**:
+    - **Scenario**: You’re migrating a database’s schema between clusters (e.g., on-premises to cloud) using `SCHEMA_ONLY` (page 150), and you want to reorganize storage locations simultaneously.
+    - **Use**: A Warehouse Plan can redefine the database’s storage locations (e.g., from `/data/my_db` to `s3a://mybucket/my_db`). Since `SCHEMA_ONLY` doesn’t move data, pair it with the [`-dc|--distcp` option](cli-options.md) to generate `distcp` plans for separately migrating the data to the new locations.
+    - **Example**:
+      ```yaml
+      databaseWarehousePlans:
+        my_db:
+          EXTERNAL_TABLE: "/my_db/ext"
+          MANAGED_TABLE: "/my_db/managed"
+      ```
+      Run: `hms-mirror -d SCHEMA_ONLY -db my_db -dc` to get the schema migration and a `distcp` plan. Again, use the `transfer.targetNamespace` to set the `s3` location. EG `s3a://mybucket`.
+
+3. **Migrating and Moving Data with SQL Strategy (SQL)**:
+    - **Scenario**: You’re using the [`SQL` data strategy](SQL.md) to migrate both metadata and data between linked clusters, and you need to adjust storage locations.
+    - **Use**: Define a Warehouse Plan to map the database’s locations to the target cluster’s storage (e.g., from `hdfs://source/my_db` to `hdfs://target/my_db`). The SQL strategy will move the data to the new locations as part of the migration.
+    - **Example**:
+      ```yaml
+      databaseWarehousePlans:
+        my_db:
+          EXTERNAL_TABLE: "/my_db/ext"
+          MANAGED_TABLE: "/my_db/managed"
+      ```
+      Run: `hms-mirror -d SQL -db my_db`.
+
+4. **Supporting Data Strategies Without Data Movement**:
+    - **Scenario**: You’re using a strategy like [`SCHEMA_ONLY`](SCHEMA_ONLY.md) that doesn’t move data, but you need a plan to migrate the data later.
+    - **Use**: Warehouse Plans map the database’s locations, and the `-dc|--distcp` option generates `distcp` scripts to handle the data migration separately, aligning with the mapped locations.
+    - **Example**: For `SCHEMA_ONLY`:
+      ```bash
+      hms-mirror -d SCHMEA_ONLY -db my_db -dc -wps my_db=/my_db/ext:/my_db/managed
+      ```
+      The Warehouse Plan ensures the dump reflects the intended target locations, and `distcp` plans are provided.
+
+5. **Consistency Within a Database Across Table Types**:
+    - **Scenario**: A database contains both External and Managed tables with various locations, and you want a unified storage structure on the target.
+    - **Use**: A Warehouse Plan reviews all locations in the database metadata and maps them to consistent target locations, regardless of table type. This is more precise than global mappings, as it’s tailored to the database.
+    - **Example**: Mapping all tables in `sales_db` to a single storage layer:
+      ```yaml
+      databaseWarehousePlans:
+        sales_db:
+          EXTERNAL_TABLE: "/new_storage/division_ext"
+          MANAGED_TABLE: "/new_storage/division_mngd"
+      ```
+
+### How to Use Warehouse Plans
+
+To implement Warehouse Plans, configure them in the `default.yaml` file under the `databaseWarehousePlans` section at the database level.
+
+Here’s how, based on the documentation and your clarification:
+
+- **Syntax**:
+  ```yaml
+  databaseWarehousePlans:
+    <database_name>:
+      EXTERNAL_TABLE: "<target_location_for_external>"
+      MANAGED_TABLE: "<target_location_for_managed>"
+  ```
+    - The metadata for `<database_name>` is analyzed, and all locations (for External and Managed tables) are mapped to the specified targets. These mappings feed into the Global Location Maps used internally.
+    - NOTE: The database name is appended to the location you specify, so do NOT include that in the location.  This allows you to use the same location for multiple databases in the scenario you want to have multiple databases share the same root location.
+
+- **Command-Line Integration**: While Warehouse Plans are configuration-driven, they work with strategies like `STORAGE_MIGRATION`, `SCHEMA_ONLY`, or `SQL`. Use `-dc|--distcp` for strategies without data movement.
+    - The Warehouse Plans can be set in the config, via the Web UI, or via the commandline option `-wps <db=ext-dir:mngd-dir[,db=ext-dir:mngd-dir]...>`
+
+  ```
+  hms-mirror -d STORAGE_MIGRATION -db my_db -dc -wps my_db=/my_db/ext:/my_db/managed
+  ```
+  or
+  ```
+  hms-mirror -d SCHEMA_ONLY -db my_db -dc
+  ```
+
+- **Execution**: The Warehouse Plan drives the location translation process. For `STORAGE_MIGRATION`, data is moved directly (if using `DISTCP` or `SQL` as the data movement strategy, page 202). For other strategies, `-dc` ensures data migration plans are provided.
+
+### Practical Tips
+- **Database Scope**: Define Warehouse Plans per database to ensure precise mapping. Unlike the older `globalLocationMap`, they don’t apply globally unless explicitly extended via "Global Warehouse Plans."
+- **Dry-Run**: Test with a dry-run (`hms-mirror -db <db_name>`) to review the mappings in the output reports (page 111) before executing (`-e`).
+- **Storage Access**: Verify permissions for the mapped locations, especially for cross-cluster scenarios (page 69, "Linking Cluster Storage Layers").
+- **Original Intent**: While designed for `STORAGE_MIGRATION` (e.g., HDFS to Ozone), their flexibility supports broader reorganization tasks.
+
+In summary, Warehouse Plans are a database-level tool in "hms-mirror" for mapping storage locations within a database’s metadata, originally for `STORAGE_MIGRATION` (e.g., HDFS to Ozone), but also valuable for `SCHEMA_ONLY` and `SQL` strategies. They construct Global Location Maps internally, ensuring accurate location translations, and can be paired with `-dc|--distcp` for separate data movement when needed.

Writerside/topics/hms-mirror-Default-Configuration-Template.md

@@ -4,7 +4,7 @@ Use this as a template for the `default.yaml` configuration file used by the `cl
 
 
 ```yaml
-# Copyright 2024 Cloudera, Inc. All Rights Reserved.
+# Copyright 2021 Cloudera, Inc. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -29,7 +29,9 @@ transfer:
 clusters:
   LEFT:
     # Set for Hive 1/2 environments
-    legacyHive:    true
+    # Remove, see 'platformType' legacyHive:    true
+    # platform type
+    platformType: 'HDP2'
     # Is the 'Hadoop COMPATIBLE File System' used to prefix data locations for this cluster.
     # It is mainly used as the transfer location for metadata (export)
     # If the primary storage for this cluster is 'hdfs' than use 'hdfs://...'
@@ -56,15 +58,16 @@ clusters:
     # This will require the user to install the jdbc driver for the metastoreDirect in $HOME/.hms-mirror/aux_libs
     metastore_direct:
       uri: "<jdbc_url_to_metastore_db_including_db>"
-      type: MYSQL|POSTGRES|ORACLE
+      type: MYSQL|POSTRES|ORACLE
       connectionProperties:
         user: "<db_user>"
         password: "<db_password>"
       connectionPool:
         min: 3
         max: 5
   RIGHT:
-    legacyHive:    false
+    # Removed, see platformType. ...legacyHive:    false
+    platformType: 'CDP7_1'
     # Is the 'Hadoop COMPATIBLE File System' used to prefix data locations for this cluster.
     # It is mainly used to as a baseline for where "DATA" will be transfered in the
     # STORAGE stage.  The data location in the source location will be move to this
@@ -93,4 +96,5 @@ clusters:
       auto:     true
       # When a table is created, run MSCK when there are partitions.
       initMSCK: true
+
 ```

pom.xml

@@ -28,7 +28,7 @@
 
     <groupId>com.cloudera.utils.hadoop</groupId>
     <artifactId>hms-mirror</artifactId>
-    <version>2.3.1.4</version>
+    <version>2.3.1.5</version>
     <packaging>jar</packaging>
 
     <name>hms-mirror</name>

src/main/java/com/cloudera/utils/hms/mirror/cli/HmsMirrorCommandLineOptions.java

@@ -1487,7 +1487,7 @@ CommandLineRunner configWarehousePlans(HmsMirrorConfig config, @Value("${hms-mir
             log.info("warehouse-plan: {}", value);
             String[] warehouseplan = value.split(",");
 
-            if (nonNull(warehouseplan) && warehouseplan.length > 0) {
+            if (nonNull(warehouseplan)) {
                 // for each plan entry, split on '=' for db=ext_dir:mngd_dir
                 for (String plan : warehouseplan) {
                     String[] planParts = plan.split("=");

src/main/java/com/cloudera/utils/hms/mirror/connections/ConnectionState.java

@@ -22,5 +22,5 @@ public enum ConnectionState {
     INITIALIZED,
     CONNECTED,
     DISCONNECTED,
-    ERROR;
+    ERROR
 }

src/main/java/com/cloudera/utils/hms/mirror/domain/Messages.java

@@ -69,11 +69,11 @@ public List<String> getMessages() {
         List<String> messageList = new ArrayList<String>();
         for (MessageCode messageCode : MessageCode.getCodes(bitSet)) {
             if (!argMap.containsKey(messageCode.ordinal())) {
-                messageList.add(messageCode.toString() + "-->" + messageCode.getDesc());
+                messageList.add(messageCode + "-->" + messageCode.getDesc());
             } else {
                 Object[] vMap = argMap.get(messageCode.ordinal());
                 String m = MessageFormat.format(messageCode.getDesc(), vMap);
-                messageList.add(messageCode.toString() + "-->" + m);
+                messageList.add(messageCode + "-->" + m);
             }
         }
 //        String[] rtn = messageList.toArray(new String[0]);

src/main/java/com/cloudera/utils/hms/mirror/domain/WarehouseMapBuilder.java

@@ -180,7 +180,7 @@ protected Object clone() throws CloneNotSupportedException {
 
         Map<String, Warehouse> newWarehousePlan = new HashMap<>();
         for (Map.Entry<String, Warehouse> entry : warehousePlans.entrySet()) {
-            newWarehousePlan.put(entry.getKey(), (Warehouse) entry.getValue().clone());
+            newWarehousePlan.put(entry.getKey(), entry.getValue().clone());
         }
         clone.setWarehousePlans(newWarehousePlan);
 

src/main/java/com/cloudera/utils/hms/mirror/domain/support/CollectionEnum.java

@@ -22,5 +22,5 @@ public enum CollectionEnum {
     IN_PROGRESS,
     COMPLETED,
     ERRORED,
-    SKIPPED;
+    SKIPPED
 }

src/main/java/com/cloudera/utils/hms/mirror/domain/support/ConfigSection.java

@@ -24,5 +24,5 @@ public enum ConfigSection {
     LEFT_CLUSTER,
     RIGHT_CLUSTER,
     TRANSLATOR,
-    TRANSFER;
+    TRANSFER
 }

src/main/java/com/cloudera/utils/hms/mirror/domain/support/ConnectionStatus.java

@@ -24,5 +24,5 @@ public enum ConnectionStatus {
     CHECK_CONFIGURATION,
     SKIPPED,
     DISABLED,
-    UNKNOWN;
+    UNKNOWN
 }