Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add GraalVM Reachability Metadata and corresponding nativeTest for Iceberg table in HiveServer2 #31526

Merged
merged 1 commit into from
Aug 16, 2024
Merged

Add GraalVM Reachability Metadata and corresponding nativeTest for Iceberg table in HiveServer2 #31526

merged 1 commit into from
Aug 16, 2024

Conversation

linghengqian
Copy link
Member

@linghengqian linghengqian commented Jun 2, 2024 •
edited
Loading

For #29052.

Changes proposed in this pull request:

Before committing this PR, I'm sure that I have checked the following options:

  • My code follows the code of conduct of this project.
  • I have self-reviewed the commit code.
  • I have (or in comment I request) added corresponding labels for the pull request.
  • I have passed maven check locally : ./mvnw clean install -B -T1C -Dmaven.javadoc.skip -Dmaven.jacoco.skip -e.
  • I have made corresponding changes to the documentation.
  • I have added corresponding unit tests for my changes.

@linghengqian linghengqian mentioned this pull request Jun 3, 2024
Merged
6 tasks
@linghengqian linghengqian force-pushed the hive-native-test branch 18 times, most recently from 99f7063 to 02e7c95 Compare June 5, 2024 14:17
@linghengqian linghengqian force-pushed the hive-native-test branch 3 times, most recently from 7640e07 to e8bac17 Compare June 6, 2024 09:37
@linghengqian linghengqian changed the title Add GraalVM Reachability Metadata and corresponding nativeTest for Hive Add GraalVM Reachability Metadata and corresponding nativeTest for HiveServer2 Jun 6, 2024
@linghengqian linghengqian mentioned this pull request Jun 6, 2024
Open
@linghengqian linghengqian force-pushed the hive-native-test branch 2 times, most recently from b7abc5b to 959b056 Compare July 29, 2024 15:32
This was referenced Jul 30, 2024
Closed
Merged
Open
@linghengqian linghengqian mentioned this pull request Aug 3, 2024
Closed
@linghengqian linghengqian force-pushed the hive-native-test branch 3 times, most recently from 0fece25 to 52508bb Compare August 4, 2024 04:09
@linghengqian linghengqian mentioned this pull request Aug 4, 2024
Open
linghengqian
linghengqian commented Aug 4, 2024
View reviewed changes
Copy link
Member Author

@linghengqian linghengqian left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

linghengqian
linghengqian commented Aug 15, 2024
View reviewed changes
Copy link
Member Author

@linghengqian linghengqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@linghengqian linghengqian force-pushed the hive-native-test branch 2 times, most recently from 3df3899 to 1eb45d1 Compare August 16, 2024 04:09
@linghengqian linghengqian mentioned this pull request Aug 16, 2024
Merged
6 tasks
linghengqian
linghengqian commented Aug 16, 2024
View reviewed changes
Copy link
Member Author

@linghengqian linghengqian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

linghengqian
linghengqian commented Aug 16, 2024
View reviewed changes
Copy link
Member Author

@linghengqian linghengqian left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • We needed a transactional HiveServer2 for our unit tests, but the options provided by apache/hive were a bit overkill. Compared to manually creating ACID tables,
set metastore.compactor.initiator.on=true;
set metastore.compactor.cleaner.on=true;
set metastore.compactor.worker.threads=5;

set hive.support.concurrency=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

CREATE TABLE IF NOT EXISTS t_order
(
    order_id   BIGINT,
    order_type INT,
    user_id    INT    NOT NULL,
    address_id BIGINT NOT NULL,
    status     VARCHAR(50),
    PRIMARY KEY (order_id) disable novalidate
) CLUSTERED BY (order_id) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES ('transactional' = 'true');
  • I prefer to create Iceberg tables directly. Whether it is an ACID table or an iceberg table, from the perspective of the apache/shardingsphere parser, it only provides support for DML statements, and any transaction rollback operations are not supported.
set iceberg.mr.schema.auto.conversion=true;

CREATE TABLE IF NOT EXISTS t_order
(
    order_id   BIGINT,
    order_type INT,
    user_id    INT    NOT NULL,
    address_id BIGINT NOT NULL,
    status     VARCHAR(50),
    PRIMARY KEY (order_id) disable novalidate
) STORED BY ICEBERG STORED AS ORC TBLPROPERTIES ('format-version' = '2');

@linghengqian linghengqian changed the title Add GraalVM Reachability Metadata and corresponding nativeTest for HiveServer2 Add GraalVM Reachability Metadata and corresponding nativeTest for Iceberg table in HiveServer2 Aug 16, 2024
@linghengqian linghengqian force-pushed the hive-native-test branch 2 times, most recently from 4262004 to f4b2af0 Compare August 16, 2024 08:47
@linghengqian linghengqian marked this pull request as ready for review August 16, 2024 09:58
@strongduanmu strongduanmu merged commit 61c6207 into apache:master Aug 16, 2024
142 checks passed
@strongduanmu strongduanmu added this to the 5.5.1 milestone Aug 16, 2024
@linghengqian linghengqian deleted the hive-native-test branch August 16, 2024 10:01
@linghengqian linghengqian mentioned this pull request Aug 21, 2024
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@strongduanmu strongduanmu strongduanmu approved these changes

Assignees
No one assigned
Labels
db: Hive in: proxy native type: enhancement
Projects
None yet
Milestone
5.5.1
Development

Successfully merging this pull request may close these issues.
2 participants
@linghengqian @strongduanmu