Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[COST-4360] Fix default dtypes mapping for OCI #4830

Merged
merged 5 commits into from
Dec 14, 2023
Merged

Conversation

myersCody
Copy link
Contributor

Jira Ticket

COST-4360

Description

This change will set the correct dtype defaults to match how we build the trino schema. Highly likely we never ran into it during testing because those columns were always present so they never got defaulted. After seeing this I went through the other providers and they seemed to match correctly.

Testing

SHOW CREATE TABLE oci_usage_line_items;
                                 Create Table
-------------------------------------------------------------------------------
 CREATE TABLE hive.org1234567.oci_usage_line_items (
    lineitem_referenceno varchar,
    lineitem_tenantid varchar,
    lineitem_intervalusagestart timestamp(3),
    lineitem_intervalusageend timestamp(3),
    product_service varchar,
    product_resource varchar,
    product_compartmentid varchar,
    product_compartmentname varchar,
    product_region varchar,
    product_availabilitydomain varchar,
    product_resourceid varchar,
    usage_consumedquantity double,
    usage_billedquantity varchar,
    usage_consumedquantityunits varchar,
    usage_consumedquantitymeasure varchar,
    lineitem_iscorrection varchar,
    lineitem_backreferenceno varchar,
    tags varchar,
    cost_overageflag varchar,
    usage_billedquantityoverage varchar,
    cost_unitpriceoverage varchar,
    cost_productsku varchar,
    cost_skuunitdescription varchar,
    cost_mycostoverage varchar,
    cost_billingunitreadable varchar,
    product_description varchar,
    cost_unitprice varchar,
    cost_currencycode varchar,
    cost_subscriptionid varchar,
    cost_mycost double,
    source varchar,
    year varchar,
    month varchar
 )
 WITH (
    external_location = 's3a://koku-bucket/data/parquet/org1234567/OCI/usage',
    format = 'PARQUET',
    partitioned_by = ARRAY['source','year','month']
 )

Notes

...

@myersCody myersCody self-assigned this Dec 12, 2023
@myersCody myersCody added the smoke-tests pr_check will build the image and run minimal required smokes label Dec 12, 2023
Copy link

codecov bot commented Dec 12, 2023

Codecov Report

Merging #4830 (031283c) into main (bdd582d) will decrease coverage by 0.0%.
The diff coverage is n/a.

Additional details and impacted files
@@           Coverage Diff           @@
##            main   #4830     +/-   ##
=======================================
- Coverage   94.0%   94.0%   -0.0%     
=======================================
  Files        364     364             
  Lines      30206   30206             
  Branches    3599    3599             
=======================================
- Hits       28383   28380      -3     
- Misses      1162    1165      +3     
  Partials     661     661             

@myersCody myersCody marked this pull request as ready for review December 13, 2023 20:41
@myersCody myersCody requested review from a team as code owners December 13, 2023 20:41
@myersCody myersCody enabled auto-merge (squash) December 14, 2023 19:36
Copy link

sonarcloud bot commented Dec 14, 2023

Quality Gate Passed Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

@myersCody myersCody merged commit 0c71f9d into main Dec 14, 2023
10 checks passed
@myersCody myersCody deleted the fix_default_dtypes branch December 14, 2023 20:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
smoke-tests pr_check will build the image and run minimal required smokes smokes-required
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants