Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] --empty flag is not working with nested columns in spectrum #627

Open
2 tasks done
gekas93 opened this issue Sep 2, 2024 · 2 comments
Open
2 tasks done

[Bug] --empty flag is not working with nested columns in spectrum #627

gekas93 opened this issue Sep 2, 2024 · 2 comments
Labels
pkg:dbt-redshift Issue affects dbt-redshift type:bug Something isn't working as documented

Comments

@gekas93
Copy link

gekas93 commented Sep 2, 2024

Is this a new bug?

  • I believe this is a new bug
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Running dbt run --empty , we get a spectrum nested query error.

Database Error in model base_***** (models/base/*****/base_*****.sql)
  Spectrum nested query error

The compiled code is the following:

create  table
    "DATABASE"."SCHEMA"."base_*****__dbt_tmp"
  as (
    select *
from
    (select * from "DATABASE"."SCHEMA"."TABLE" where false limit 0)
  );

If we remove the limit 0 or we remove the subquery having something like this:

  create  table
    "DATABASE"."SCHEMA"."base_*****__dbt_tmp"
  as (
    select * from "DATABASE"."SCHEMA"."TABLE" where false limit 0
  );

The query works on redshift.

Expected Behavior

dbt run --empty works on redshift spectrum.

Steps To Reproduce

  1. To have some spectrum source with array column.
  2. Create some model with select * to that source.
  3. Run dbt run --empty

Relevant log output

No response

Environment

- OS: Mac
- Python:3.9.18
- dbt-core:1.8.1
- dbt-redshift:1.8.1

Additional Context

We figured out that the code that generates this line of code is in relation.py file.

 def render_limited(self) -> str:
        rendered = self.render()
        if self.limit is None:
            return rendered
        elif self.limit == 0:
            return f"(select * from {rendered} where false limit 0){self._render_limited_alias()}"
        else:
            return f"(select * from {rendered} limit {self.limit}){self._render_limited_alias()}"

If we remove limit 0, it works.

@gekas93 gekas93 added type:bug Something isn't working as documented triage:product In Product's queue labels Sep 2, 2024
@amychen1776 amychen1776 removed the triage:product In Product's queue label Sep 5, 2024
@colin-rogers-dbt colin-rogers-dbt transferred this issue from dbt-labs/dbt-adapters Sep 13, 2024
@colin-rogers-dbt
Copy link
Contributor

This is going to require a dbt-redshift specific solution as currently base adapter (nor redshift adapter) are aware of whether a source is a spectrum table

@colin-rogers-dbt
Copy link
Contributor

Likely addressing this will require users to configure their spectrum table references as sources and for dbt-redshift to check if a source is a spectrum table before rendering. We will spike out evaluating if/how we can support this

@mikealfare mikealfare added the pkg:dbt-redshift Issue affects dbt-redshift label Jan 15, 2025
@mikealfare mikealfare transferred this issue from dbt-labs/dbt-redshift Jan 23, 2025
mikealfare added a commit that referenced this issue Jan 23, 2025
* Update pre-commit-hooks requirement from ~=4.4 to ~=4.5

Updates the requirements on [pre-commit-hooks](https://github.com/pre-commit/pre-commit-hooks) to permit the latest version.
- [Release notes](https://github.com/pre-commit/pre-commit-hooks/releases)
- [Changelog](https://github.com/pre-commit/pre-commit-hooks/blob/main/CHANGELOG.md)
- [Commits](pre-commit/pre-commit-hooks@v4.4.0...v4.5.0)

---
updated-dependencies:
- dependency-name: pre-commit-hooks
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>

* Add automated changelog yaml from template for bot PR

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Github Build Bot <[email protected]>
Co-authored-by: Mike Alfare <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pkg:dbt-redshift Issue affects dbt-redshift type:bug Something isn't working as documented
Projects
None yet
Development

No branches or pull requests

4 participants