Support for hive cache table?

[repidlei]

CACHE TABLE tmp_global_cust_code xxxxx

I use a cache table and then use it later,but it will report an error that does not move the source.

org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source hdfs://namenodedev1.synnex.org:8020/warehouse/tablespace/managed/hive/dw_hygbl.db/dwd_hyve_inv_fg_aging_df/.hive-staging_hive_2021-01-04_18-39-28_226_3267340521211185321-2/-ext-10000/part-00000-38f0a35b-2479-421c-9185-5939e1e54981-c000 to destination hdfs://namenodedev1.synnex.org:8020/warehouse/tablespace/managed/hive/dw_hygbl.db/dwd_hyve_inv_fg_aging_df/date_flag=2020-12-29/part-00000-38f0a35b-2479-421c-9185-5939e1e54981-c000;
	at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:106)
	at org.apache.spark.sql.hive.HiveExternalCatalog.loadPartition(HiveExternalCatalog.scala:843)
	at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.processInsert(InsertIntoHiveTable.scala:248)
	at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.run(InsertIntoHiveTable.scala:99)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:104)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:102)
	at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:115)
	at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
	at org.apache.spark.sql.Dataset$$anonfun$6.apply(Dataset.scala:190)
	at org.apache.spark.sql.Dataset$$anonfun$52.apply(Dataset.scala:3259)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:77)
	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3258)
	at org.apache.spark.sql.Dataset.<init>(Dataset.scala:190)
	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:75)
	at org.apache.spark.sql.SparkSQLUtils$.toDataFrame(SparkSQLUtils.scala:39)
	at yaooqinn.kyuubi.operation.statement.ExecuteStatementInClientMode.execute(ExecuteStatementInClientMode.scala:148)
	at yaooqinn.kyuubi.operation.statement.ExecuteStatementOperation$$anon$1$$anon$2.run(ExecuteStatementOperation.scala:74)
	at yaooqinn.kyuubi.operation.statement.ExecuteStatementOperation$$anon$1$$anon$2.run(ExecuteStatementOperation.scala:70)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at yaooqinn.kyuubi.operation.statement.ExecuteStatementOperation$$anon$1.run(ExecuteStatementOperation.scala:70)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

How should this be handled?
version: kyuubi-0.8.0-SNAPSHOT-bin-spark-2.1.3 spark-2.3.2

[yaooqinn]

Can you just check the error instead？ It said 'unable to move'

[repidlei]

Can you just check the error instead？ It said 'unable to move'

I tested this SQL with Hive and it worked fine.I wonder if Hive and Spark have different mechanisms for cache tables.Hive and Kyuubi use the same account, there will be no access problem.

[yaooqinn]

you can follow the stack trace to find the root cause

[pan3793]

@leixun233 It does not seem to be a Kyuubi related issue, you can use spark-shell to directly test SQL without Kyuubi to see if it can reproduce

run spark-shell

spark.sql(""" PUT YOUR SQL HERE  """).show(false)