Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query never sent to server if it contains set statements #238

Open
lucastheisen opened this issue Apr 2, 2024 · 3 comments
Open

Query never sent to server if it contains set statements #238

lucastheisen opened this issue Apr 2, 2024 · 3 comments

Comments

@lucastheisen
Copy link

I have a query that is failing due to memory kill of the form:

select
  *
from my_schema.my_table
where
  date >= '2024-03-12T00:00:00Z'
  date < '2024-03-19T00:00:00Z'
order by date asc;

However, when i try to increase memory:

SET mapreduce.map.memory.mb=5120;
SET mapreduce.reduce.memory.mb=5120;
SET mapreduce.map.java.opts=-Xmx4608m;
SET mapreduce.reduce.java.opts=-Xmx4608m;
select
  *
from my_schema.my_table
where
  date >= '2024-03-12T00:00:00Z'
  date < '2024-03-19T00:00:00Z'
order by date asc;

The client simply exits with EOF being the error. Running the same query on ambari works.

@beltran
Copy link
Owner

beltran commented Apr 4, 2024

What is the output of the first query? When you say the output of the second one is EOF you mean the client crashes and EOF gets printed (just those three chars) and the client exists? What do you get when you run the query in beeline?

@lucastheisen
Copy link
Author

@beltran ,

What is the output of the first query? When you say the output of the second one is EOF you mean the client crashes and EOF gets printed (just those three chars) and the client exists?

Correct. The client exits and the error message EOF is printed (because i let the error object bubble out to my cli).

What do you get when you run the query in beeline?

Not sure... I've never used beeline. But also, i had to refactor my query to make better use of partition pruning and smaller extract sizes to move forward so increasing memory is no longer a blocker for me. You can close this issue if you want, but still would be good to know if using SET statements is not-supported by this driver, or if maybe i was doing it wrong (the later is not at all unlikely).

@beltran
Copy link
Owner

beltran commented Apr 14, 2024

They are supported, see this test. But I don't know why it wasn't working for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants