Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Support Athena Adapter #251

Open
GabrielEisenbergOlympus opened this issue Oct 16, 2024 · 1 comment
Open

[FEATURE] Support Athena Adapter #251

GabrielEisenbergOlympus opened this issue Oct 16, 2024 · 1 comment
Assignees
Labels
feature This is is requesting a new feature

Comments

@GabrielEisenbergOlympus
Copy link

GabrielEisenbergOlympus commented Oct 16, 2024

Is your feature request related to a problem? Please describe.
I am using the dbt-athena adapter which was recently taken over by dbt-labs and was announced as being GA. AutomateDV complains that it does not have the three following macros:

  1. athena__cast_date
  2. athena__cast_datetime
  3. athena__get_escape_characters

Describe the solution you'd like
Add these macros to the project. They seem relatively easy to add. As I am new to the project, I might be missing the nuances and complexities of adding a new adapter.

Describe alternatives you've considered
None

Additional context
The Athena escape character is a double quote.

The casting can be done along the lines of:

date(date_parse('2022-10-20', '%Y-%m-%d'))
date_parse('2022-10-20 05:00:01', '%Y-%m-%d %h:%i:%s')
from_iso8601_timestamp('2022-10-20T05:00:01.000Z')

I'm waiting on approval to contribute back but if someone can do this sooner than later, please feel free! 😄

@GabrielEisenbergOlympus GabrielEisenbergOlympus added the feature This is is requesting a new feature label Oct 16, 2024
@GabrielEisenbergOlympus
Copy link
Author

GabrielEisenbergOlympus commented Oct 16, 2024

As an attempt I did the following. Obviously, feel free to modify/correct the code.

At macros/internal/metadata_processing/get_escape_characters.sql, I added:

{%- macro athena__get_escape_characters() %}
    {%- do return (('"', '"')) -%}
{%- endmacro %}

At macros/supporting/casting/cast_date.sql, I added:

{%- macro athena__cast_date(column_str, as_string=false, alias=none) -%}
    {%- if not as_string -%}
        date_parse({{ column_str }}, '%Y-%m-%d')
    {%- else -%}
        date_parse('{{ column_str }}', '%Y-%m-%d')
    {%- endif -%}
    {%- if alias %} AS {{ alias }} {%- endif %}
{%- endmacro -%}

At macros/supporting/casting/cast_datetime.sql, I added:

{%- macro athena__cast_datetime(column_str, as_string=false, alias=none, date_type=none) -%}
    from_iso8601_timestamp({{ column_str }})
    {%- if alias %} AS {{ alias }} {%- endif %}
{%- endmacro -%}

At macros/supporting/data_types/type_binary.sql, I added:

{%- macro athena__type_binary(for_dbt_compare=false) -%}
    varbinary
{%- endmacro -%}

At "macros/supporting/hash_components/select_hash_alg.sql", I added:

{% macro athena__hash_alg_md5() -%}

    {% do return("md5(to_utf8([HASH_STRING_PLACEHOLDER]))") %}

{% endmacro %}

{% macro athena__hash_alg_sha1() -%}

    {% do return("sha1(to_utf8([HASH_STRING_PLACEHOLDER]))") %}

{% endmacro %}

{% macro athena__hash_alg_sha256() -%}

    {% do return("sha2(to_utf8([HASH_STRING_PLACEHOLDER]), 256)") %}

{% endmacro %}

At macros/supporting/hash_components/standard_column_wrapper.sql, I added:

{%- macro athena__standard_column_wrapper(hash_content_casing) -%}

    {%- if hash_content_casing == 'upper' -%}
        {%- set standardise -%}
            COALESCE(UPPER(TRIM(CAST([EXPRESSION] AS {{ automate_dv.type_string() }}))), '')
        {%- endset -%}
    {%- else -%}
        {%- set standardise -%}
            COALESCE(TRIM(CAST([EXPRESSION] AS {{ automate_dv.type_string() }})), '')
        {%- endset -%}
    {%- endif -%}

    {% do return(standardise) -%}

{%- endmacro -%}

At macros/staging/null_columns.sql, I added:

{%- macro athena__null_column_sql(col_name, default_value) -%}

    {{ col_name }} AS {{ col_name ~ "_ORIGINAL" }},
    COALESCE({{ col_name }}, '{{ default_value }}') AS {{ col_name }}

{%- endmacro -%}

At macros/supporting/hash_components/null_expression.sql, I added:

{%- macro athena__null_expression(standardise, column_str, null_placeholder_string) -%}

    {%- set column_expression -%}
        COALESCE({{ standardise | replace('[EXPRESSION]', column_str) }}, '{{ null_placeholder_string }}')
    {%- endset -%}

    {% do return(column_expression) %}

{%- endmacro -%}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature This is is requesting a new feature
Projects
None yet
Development

No branches or pull requests

2 participants