[Bug] The Kyuubi query did not take effect when entering the ERROR_STATE from RUNNING-STATE

[beliefer]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

The Kyuubi query did not take effect when entering the ERROR_STATE from RUNNING-STATE, and eventually changed back to RUNNING-STATE. Then keep running continuously until timeout or manual kill. The log show below.

: 26/01/25 04:24:11 INFO[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Processing g10_us's query[284c07fb-a061-4489-a725-e3d68b2174d9]: RUNNING_STATE -> ERROR_STATE, time taken: 7.146 seconds"

: 26/01/25 04:24:11 ERROR[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Exception in query[284c07fb-a061-4489-a725-e3d68b2174d9]: Error operating ExecuteStatement:[ org.apache.hadoop.mapred.InvalidInputException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.hadoop.mapred.InvalidInputException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124

: Caused by:[ java.io.IOException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Fjava.io.IOException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124

2026-01-25 04:24:13.942 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:18.943 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:23.944 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE

The last log message is repeatedly appearing.

Affects Version(s)

1.8.0

Kyuubi Server Log Output

: 26/01/25 04:24:11 INFO[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Processing g10_us's query[284c07fb-a061-4489-a725-e3d68b2174d9]: RUNNING_STATE -> ERROR_STATE, time taken: 7.146 seconds"

: 26/01/25 04:24:11 ERROR[ org.apache.kyuubi.engine.spark.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.engine.spark.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) "Exception in query[284c07fb-a061-4489-a725-e3d68b2174d9]: Error operating ExecuteStatement:[ org.apache.hadoop.mapred.InvalidInputException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.hadoop.mapred.InvalidInputException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124

: Caused by:[ java.io.IOException:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Fjava.io.IOException%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Input path does not exist: hdfs:///src/day/dt=20260124

2026-01-25 04:24:13.942 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:18.943 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE
2026-01-25 04:24:23.944 INFO KyuubiSessionManager-exec-pool: Thread-3661628[ org.apache.kyuubi.operation.ExecuteStatement:](https://wx2.qq.com/cgi-bin/mmwebwx-bin/webwxcheckurl?requrl=http%3A%2F%2Forg.apache.kyuubi.operation.ExecuteStatement%3A&skey=%40crypt_11a8b7bc_35bc0fdd7c3f497d64e3e53a5fb58e7e&deviceid=e796872855946416&pass_ticket=j9EBOJRwhyStCHb2HootXAMkM7quWMZ1Ddq1tGgilJa2AjX5%252Fnq9HSwPdoYTOxZYr66sJhtnFExRF9PaVCy6rw%253D%253D&opcode=2&scene=1&username=@59856e166d674e96884ad5707b6ab13f218a37e3e0aaae62c0c66df18cadde6e) Query[284c07fb-a061-4489-a725-e3d68b2174d9] in RUNNING_STATE

Kyuubi Engine Log Output

Kyuubi Server Configurations

spark.kyuubi.authentication.rest.bypass.url.pattern   /api/v1/admin/engineMetrics
spark.kyuubi.backend.engine.exec.pool.size   600
spark.kyuubi.backend.engine.exec.pool.wait.queue.size 200
spark.kyuubi.backend.server.exec.pool.size   600
spark.kyuubi.backend.server.exec.pool.wait.queue.size 200
spark.kyuubi.client.ipAddress 7.36.133.145
spark.kyuubi.credentials.hadoopfs.uris hdfs://***
spark.kyuubi.engine.credentials  
spark.kyuubi.engine.engineLog.path  /home/hadoop/logs/kyuubi18/***
spark.kyuubi.engine.event.loggers   CUSTOM,SPARK
spark.kyuubi.engine.metrics.internal.api.path   /metrics/prometheus/sep
spark.kyuubi.engine.share.level.subdomain ***
spark.kyuubi.engine.submit.time  1769285944714
spark.kyuubi.engine.yarn.submit.timeout   PT30M
spark.kyuubi.engineEnv.GDC_KYUUBI_USER g10_us
spark.kyuubi.frontend.protocols  THRIFT_BINARY,REST
spark.kyuubi.frontend.thrift.max.worker.threads 5000
spark.kyuubi.frontend.thrift.min.worker.threads 200
spark.kyuubi.ha.addresses  ***
spark.kyuubi.ha.enabled true
spark.kyuubi.ha.engine.lock.path /***
spark.kyuubi.ha.engine.ref.id db340152-1d8b-473d-b600-1053aa400dc8
spark.kyuubi.ha.namespace  /***
spark.kyuubi.ha.zookeeper.auth.keytab  ***
spark.kyuubi.ha.zookeeper.auth.principal  ***
spark.kyuubi.ha.zookeeper.auth.type KERBEROS
spark.kyuubi.ha.zookeeper.client.port  2191
spark.kyuubi.ha.zookeeper.engine.auth.type   KERBEROS
spark.kyuubi.ha.zookeeper.session.timeout 600000
spark.kyuubi.metrics.engine.prometheus.enable   true
spark.kyuubi.metrics.engine.prometheus.path  /engine_metrics/*
spark.kyuubi.metrics.host.enabled   true
spark.kyuubi.metrics.prometheus.port   47070
spark.kyuubi.metrics.reporters   PROMETHEUS
spark.kyuubi.server.ipAddress 10.90.133.157
spark.kyuubi.session.connection.url ***:37070
spark.kyuubi.session.engine.idle.timeout  PT5M
spark.kyuubi.session.engine.initialize.timeout  PT30M
spark.kyuubi.session.engine.spark.showProgress  true
spark.kyuubi.session.idle.timeout   PT180M
spark.kyuubi.session.real.user   hive
spark.kyuubi.sql.engine.metrics.path   /metrics/prometheus

Kyuubi Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[github-actions]

Hello @beliefer,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.