Skip to content

Commit 484f20f

Browse files
dstreevdstreev
committed
Metastore Direct Process to cleanup partition/fs discrepancies. #44 and #42
Display Sorting. Oracle table_catagory_count issue: #48 and #47
1 parent 695ebe1 commit 484f20f

28 files changed

+1548
-306
lines changed

‎README.md‎

Lines changed: 29 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -4,32 +4,41 @@ This application has 3 sub-programs:
44

55
- [`sre`](./sre.md) is used to find potential 'Hive' performance issues caused by small files and excessive partitions.
66
- [`u3`](./u3.md) is used to review 'Hive 1/2' environments for Hive3 upgrade planning.
7+
- **NEW** [`u3e`](./u3e.md) is used to 'execute' the upgrade scripts for Hive3 directly against the metastore database. Run against the metastore DB **AFTER** the upgrade to Hive 3.
78
- [`perf`](./perf.md) is used to check the throughput of a JDBC connection.
89

910
### [Trouble-Shooting](./troubleshooting.md)
1011

1112
## Supported Metastore DB's
1213

13-
| Sub-Program | Database | Version | Tested | Notes |
14-
|:---|:---|:---|:---------------------------------|:---------------------------------------------------------------------------------------------------------------------------------|
15-
| `u3` | MySql | 5.6 | Limited | Recommend upgrading 5.7. This is the lower MySql supported env for HDP |
16-
| | | 5.7 | Yes | |
17-
| | | 8.0 | No | Not supported by HDP |
18-
| | MariaDb | 10.1 | No, but should work as 10.2 does | |
19-
| | | 10.2 | Yes | |
20-
| | Postgresql | 9.6 | No, but should work | |
21-
| | | 10 | Yes | Field Tested, May still be a few rough edges |
22-
| | | 11 | No, but should work at 10 does | |
23-
| | Oracle | 12 | Yes | Field Tested, May still be a few rough edges |
24-
| `sre` | MySql | 5.6 | Limited | Recommend upgrading 5.7. This is the lower MySql supported env for HDP |
25-
| | | 5.7 | Partly | Some `sre` reports use CTE in the SQL, which isn't supported in this version. Those report will error, the other will run fine. |
26-
| | | 8.0 | No | Not supported by HDP |
27-
| | MariaDb | 10.1 | No, but should work as 10.2 does | |
28-
| | | 10.2 | Yes | |
29-
| | Postgresql | 9.6 | No, but should work | |
30-
| | | 10 | Yes | Field Tested, May still be a few rough edges |
31-
| | | 11 | No, but should work at 10 does | |
32-
| | Oracle | 12 | Yes | Field Tested, May still be a few rough edges |
14+
15+
| Sub-Program | Database | Version | Tested | Notes |
16+
|:------------|:----------------|:--------|:---------------------------------|:---------------------------------------------------------------------------------------------------------------------------------|
17+
| `u3` | MySql | 5.6 | Limited | Recommend upgrading 5.7. This is the lower MySql supported env for HDP |
18+
| | | 5.7 | Yes | |
19+
| | | 8.0 | No | Not supported by HDP |
20+
| | MariaDb | 10.1 | No, but should work as 10.2 does | |
21+
| | | 10.2 | Yes | |
22+
| | Postgresql | 9.6 | No, but should work | |
23+
| | | 10 | Yes | Field Tested, May still be a few rough edges |
24+
| | | 11 | No, but should work at 10 does | |
25+
| | Oracle | 12 | Yes | Field Tested, May still be a few rough edges |
26+
| `u3e` | MySql | 5.6 | Limited | Recommend upgrading 5.7. This is the lower MySql supported env for HDP |
27+
| | | 5.7 | Yes | |
28+
| | | 8.0 | No | Not supported by HDP |
29+
| | MariaDb | 10.1 | No, but should work as 10.2 does | |
30+
| | | 10.2 | Yes | |
31+
| | Postgresql | * | NOT YET IMPLEMENTED | |
32+
| | Oracle | * | NOT YET IMPLEMENTED | |
33+
| `sre` | MySql | 5.6 | Limited | Recommend upgrading 5.7. This is the lower MySql supported env for HDP |
34+
| | | 5.7 | Partly | Some `sre` reports use CTE in the SQL, which isn't supported in this version. Those report will error, the other will run fine. |
35+
| | | 8.0 | No | Not supported by HDP |
36+
| | MariaDb | 10.1 | No, but should work as 10.2 does | |
37+
| | | 10.2 | Yes | |
38+
| | Postgresql | 9.6 | No, but should work | |
39+
| | | 10 | Yes | Field Tested, May still be a few rough edges |
40+
| | | 11 | No, but should work at 10 does | |
41+
| | Oracle | 12 | Yes | Field Tested, May still be a few rough edges |
3342

3443
Ensure you have the database appropriate driver in the `${HOME}/.hive-sre/aux_libs` directory.
3544

‎bin/setup.sh‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,8 @@ rm -f $BASE_DIR/bin/*
4545

4646
cp -f hive-sre $BASE_DIR/bin
4747
cp -f hive-sre-cli $BASE_DIR/bin
48+
cp -f u3/get_part_ids.sh $BASE_DIR/bin
49+
cp -f u3/mysql_missing_parts.sh $BASE_DIR/bin
4850

4951
if [ -f hive-sre-shaded.jar ]; then
5052
cp -f hive-sre-shaded.jar $BASE_DIR/lib
@@ -57,15 +59,21 @@ fi
5759
chmod -R +r $BASE_DIR
5860
chmod +x $BASE_DIR/bin/hive-sre
5961
chmod +x $BASE_DIR/bin/hive-sre-cli
62+
chmod +x $BASE_DIR/bin/get_part_ids.sh
63+
chmod +x $BASE_DIR/bin/mysql_missing_parts.sh
6064

6165
if (( $EUID == 0 )); then
6266
echo "Setting up global links"
6367
ln -sf $BASE_DIR/bin/hive-sre /usr/local/bin/hive-sre
6468
ln -sf $BASE_DIR/bin/hive-sre-cli /usr/local/bin/hive-sre-cli
69+
ln -sf $BASE_DIR/bin/get_part_ids.sh /usr/local/bin/get_part_ids.sh
70+
ln -sf $BASE_DIR/bin/mysql_missing_parts.sh /usr/local/bin/mysql_missing_parts.sh
6571
else
6672
mkdir -p $HOME/bin
6773
ln -sf $BASE_DIR/bin/hive-sre $HOME/bin/hive-sre
6874
ln -sf $BASE_DIR/bin/hive-sre-cli $HOME/bin/hive-sre-cli
75+
ln -sf $BASE_DIR/bin/get_part_ids.sh $HOME/bin/get_part_ids.sh
76+
ln -sf $BASE_DIR/bin/mysql_missing_parts.sh $HOME/bin/mysql_missing_parts.sh
6977
echo "Executable in $HOME/bin . Add this to the environment path."
7078
fi
7179

0 commit comments

Comments
 (0)