HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited #4851

basapuram-kumar · 2023-11-03T17:24:54Z

What changes were proposed in this pull request?

As per this jira HIVE-27847 , this fix will prevent query failures for Casting NUMERIC types to TIMESTAMP.

Why are the changes needed?

To avoid query failures on master branch, 4.0.0-alpha-1 & 2 branches.

Does this PR introduce any user-facing change?

No

Is the change a dependency upgrade?

No

How was this patch tested?

Reproducible steps are added in the jira HIVE-27847, after applying this fix, the failed query gets succeded.

aturoczy

Could you please create a test around it? The ticket is well described what it is about. But would be great to have test(s) to check what are the new results. Depend of the input

sonarcloud · 2023-11-11T14:22:21Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

github-actions · 2024-01-11T00:20:12Z

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the [email protected] list if the patch is in need of reviews.

okumin · 2024-05-14T07:12:01Z

@basapuram-kumar @deniskuzZ Hi, we are also hitting this problem, and I found the issue happens when both of the following conditions are satisfied.

GenericUDFTimestamp is evaluated at runtime where no hive-site.xml is located, which typically means TezChild
It is vectorized. VectorSelectOperator is unlikely to have the issue

I guess the current implementation partially works but it could not be perfect as intToTimestampInSeconds can be still wrongly configured. Could you please try to fix the issue in the correct way? We can take it over if you prefer it.

This is the minimal reproduction in my environment.

CREATE TABLE test AS SELECT 0 AS ts;

set hive.fetch.task.conversion=none;
set hive.strict.timestamp.conversion=false;
set hive.vectorized.execution.enabled=false;

SELECT CAST(ts AS TIMESTAMP) FROM test;

Caused by: java.lang.RuntimeException: Map operator initialization failed
	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:351)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:292)
	... 16 more
Caused by: org.apache.hadoop.hive.ql.exec.UDFArgumentException: Casting NUMERIC types to TIMESTAMP is prohibited (hive.strict.timestamp.conversion)
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDFTimestamp.initialize(GenericUDFTimestamp.java:91)
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:149)
	at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:184)
	at org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:1073)
	at org.apache.hadoop.hive.ql.exec.Operator.initEvaluatorsAndReturnStruct(Operator.java:1099)
	at org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:74)
	at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:360)

okumin · 2024-05-15T02:42:28Z

Also, GenericUDFTimestamp.java could not be the only file that accesses runtime configurations. We should review all UDFs accessing HiveConf.

zratkai · 2024-05-15T09:03:59Z

ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java

    intToTimestampInSeconds = SessionState.get() != null ? SessionState.get().getConf()
        .getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS) : new HiveConf()
        .getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS);

-    if (strict) {
+    SessionState ss = SessionState.get();


Please use more convenient names like sessionState here.

zratkai · 2024-05-15T09:07:47Z

ql/src/test/queries/clientpositive/non_strict_numeric_to_timestamp_conversion.q

@@ -0,0 +1,12 @@
+set hive.strict.timestamp.conversion=false;


I can not see the q.out file for this test in this commit.

zratkai

Please check my comments!

okumin · 2024-05-15T11:07:47Z

ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java

-    if (strict) {
+    SessionState ss = SessionState.get();
+
+    if (ss != null && ss.getConf().getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION)) {


The root cause is SessionState is not available at runtime. So, we need to inject the proper config from anywhere else.
For example, we implement GenericUDF#configure for some UDFs in case that SessionState is unavailable.

hive/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToUnixTimeStamp.java

Lines 125 to 132 in a1420ed

@Override

public void configure(MapredContext context) {

if (context != null) {

formatter = InstantFormatter.ofConfiguration(context.getJobConf());

String timeZoneStr = HiveConf.getVar(context.getJobConf(), HiveConf.ConfVars.HIVE_LOCAL_TIME_ZONE);

timeZone = TimestampTZUtil.parseTimeZone(timeZoneStr);

}

}

hive/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFRegExp.java

Lines 64 to 70 in a1420ed

@Override

public void configure(MapredContext context) {

if (context != null) {

if(HiveConf.getBoolVar(context.getJobConf(), HiveConf.ConfVars.HIVE_USE_GOOGLE_REGEX_ENGINE)){

this.useGoogleRegexEngine=true;

}

}

okumin · 2024-05-15T11:08:11Z

ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java


  @Override
  public ObjectInspector initialize(ObjectInspector[] arguments) throws UDFArgumentException {
    checkArgsSize(arguments, 1, 1);
    checkArgPrimitive(arguments, 0);
    checkArgGroups(arguments, 0, tsInputTypes, STRING_GROUP, DATE_GROUP, NUMERIC_GROUP, VOID_GROUP, BOOLEAN_GROUP);

-    strict = SessionState.get() != null ? SessionState.get().getConf()
-        .getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION) : new HiveConf()
-        .getBoolVar(ConfVars.HIVE_STRICT_TIMESTAMP_CONVERSION);
    intToTimestampInSeconds = SessionState.get() != null ? SessionState.get().getConf()
        .getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS) : new HiveConf()
        .getBoolVar(ConfVars.HIVE_INT_TIMESTAMP_CONVERSION_IN_SECONDS);


I guess this also needs to be correctly configured.

dengzhhu653 · 2024-05-20T08:42:53Z

Hello @basapuram-kumar, thank you for the contribution, could you please address the suggested comments if have cycles? Thank you in advance!

dengzhhu653 · 2024-06-03T00:27:52Z

Would anyone be willing to take over this PR since we have no heard from basapuram-kumar? cc @okumin @zratkai

okumin · 2024-06-03T07:36:17Z

So, let me take a glance.

okumin · 2024-06-05T00:44:44Z

@dengzhhu653 I created a new PR because I have no permission to directly update @basapuram-kumar 's branch.
#5278

HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited

acb56f9

asf-ci-hive added tests pending tests passed and removed tests pending labels Nov 3, 2023

aturoczy suggested changes Nov 8, 2023

View reviewed changes

HIVE-27847: Adding test cases

3c4fb4f

asf-ci-hive added tests pending and removed tests passed labels Nov 9, 2023

basapuram-kumar requested a review from aturoczy November 9, 2023 13:18

asf-ci-hive added tests failed and removed tests pending labels Nov 10, 2023

HIVE-27847: Adding test cases-v2

c7fedd1

asf-ci-hive added tests pending and removed tests failed labels Nov 11, 2023

asf-ci-hive added tests failed and removed tests pending labels Nov 11, 2023

github-actions bot added the stale label Jan 11, 2024

github-actions bot closed this Jan 18, 2024

deniskuzZ removed the stale label Apr 6, 2024

deniskuzZ reopened this Apr 6, 2024

asf-ci-hive added tests pending tests failed and removed tests failed tests pending labels Apr 6, 2024

zratkai reviewed May 15, 2024

View reviewed changes

zratkai suggested changes May 15, 2024

View reviewed changes

okumin reviewed May 15, 2024

View reviewed changes

dengzhhu653 closed this Jun 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited #4851

HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited #4851

basapuram-kumar commented Nov 3, 2023

aturoczy left a comment

sonarcloud bot commented Nov 11, 2023

github-actions bot commented Jan 11, 2024

okumin commented May 14, 2024 •

edited

Loading

okumin commented May 15, 2024 •

edited

Loading

zratkai May 15, 2024

zratkai May 15, 2024

zratkai left a comment

okumin May 15, 2024

okumin May 15, 2024

dengzhhu653 commented May 20, 2024

dengzhhu653 commented Jun 3, 2024

okumin commented Jun 3, 2024

okumin commented Jun 5, 2024

	@Override
	public void configure(MapredContext context) {
	if (context != null) {
	formatter = InstantFormatter.ofConfiguration(context.getJobConf());
	String timeZoneStr = HiveConf.getVar(context.getJobConf(), HiveConf.ConfVars.HIVE_LOCAL_TIME_ZONE);
	timeZone = TimestampTZUtil.parseTimeZone(timeZoneStr);
	}
	}

HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited #4851

HIVE-27847: Casting NUMERIC types to TIMESTAMP is prohibited #4851

Conversation

basapuram-kumar commented Nov 3, 2023

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

Is the change a dependency upgrade?

How was this patch tested?

aturoczy left a comment

Choose a reason for hiding this comment

sonarcloud bot commented Nov 11, 2023

github-actions bot commented Jan 11, 2024

okumin commented May 14, 2024 • edited Loading

okumin commented May 15, 2024 • edited Loading

zratkai May 15, 2024

Choose a reason for hiding this comment

zratkai May 15, 2024

Choose a reason for hiding this comment

zratkai left a comment

Choose a reason for hiding this comment

okumin May 15, 2024

Choose a reason for hiding this comment

okumin May 15, 2024

Choose a reason for hiding this comment

dengzhhu653 commented May 20, 2024

dengzhhu653 commented Jun 3, 2024

okumin commented Jun 3, 2024

okumin commented Jun 5, 2024

okumin commented May 14, 2024 •

edited

Loading

okumin commented May 15, 2024 •

edited

Loading