Skip to content

feat: Support pre-aggregation-specific data source credentials (CORE-…#10587

Merged
ovr merged 16 commits intomasterfrom
feat/separate-pre-aggregations-driver
Apr 10, 2026
Merged

feat: Support pre-aggregation-specific data source credentials (CORE-…#10587
ovr merged 16 commits intomasterfrom
feat/separate-pre-aggregations-driver

Conversation

@ovr
Copy link
Copy Markdown
Member

@ovr ovr commented Mar 31, 2026

No description provided.

@github-actions github-actions bot added driver:mongodb Issues relating to the MongoBI driver driver:redshift Issues relating to the Redshift driver driver:mysql Issues relating to the MySQL/MariaDB driver driver:bigquery Issues related to the BigQuery driver driver:snowflake Issues relating to the Snowflake driver driver:clickhouse Issues related to the ClickHouse driver driver:athena Issues related to the AWS Athena driver driver:mssql Issues relating to the MSSQL driver driver:oracle Issues relating to the Oracle driver driver:prestodb Issues relating to the PrestoDB driver driver:postgres Issues relating to the Postgres driver driver:hive Issues relating to the Apache Hive driver driver:mysql-aurora-serverless Issues relating to the AWS MySQL Aurora Serverless driver driver:sqlite Issues relating to the SQLite driver driver:druid Issues relating to the Druid driver driver:elasticsearch Issues relating to the ElasticSearch driver driver:dremio driver:questdb javascript Pull requests that update Javascript code driver:firebolt data source driver driver:databricks driver:duckdb driver:pinot cubejs-jdbc-driver labels Mar 31, 2026
@ovr ovr force-pushed the feat/separate-pre-aggregations-driver branch from 9ab1520 to 141cdad Compare March 31, 2026 14:37
@ovr ovr force-pushed the feat/separate-pre-aggregations-driver branch from 141cdad to 05bc016 Compare March 31, 2026 15:30
Comment on lines +127 to +136
if (preAggregations) {
const dsMatch = key.match(/^(CUBEJS_DS_[A-Z0-9_]+?_)(DB_|JDBC_|AWS_|DATABASE|FIREBOLT_)(.*)/);
if (dsMatch) {
return `${dsMatch[1]}PRE_AGGREGATIONS_${dsMatch[2]}${dsMatch[3]}`;
}

if (key.startsWith('CUBEJS_')) {
return key.replace(/^CUBEJS_/, 'CUBEJS_PRE_AGGREGATIONS_');
}
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Consider adding a unit test for edge cases in the regex. The regex ^(CUBEJS_DS_[A-Z0-9_]+?_)(DB_|JDBC_|AWS_|DATABASE|FIREBOLT_)(.*) uses a non-greedy match for the DS name which could behave unexpectedly with certain datasource names.

For example, a datasource named db would produce the key CUBEJS_DS_DB_DB_HOST. The non-greedy [A-Z0-9_]+?_ would match CUBEJS_DS_D first, then try B_DB_HOST against the alternation — B_ doesn't match DB_, so it extends to CUBEJS_DS_DB_ and then matches DB_HOST. This works, but edge cases like datasource names containing DB, AWS, JDBC, etc. as substrings should be tested.

Also, DATABASE in the alternation doesn't have a trailing underscore. For CUBEJS_DS_FOO_DATABASE_SECRET_ARN, group 2 captures DATABASE and group 3 captures _SECRET_ARN (including the leading underscore). This produces CUBEJS_DS_FOO_PRE_AGGREGATIONS_DATABASE_SECRET_ARN which is correct, but the asymmetry with other alternatives (which have trailing _) is subtle.

@codecov
Copy link
Copy Markdown

codecov bot commented Apr 9, 2026

Codecov Report

❌ Patch coverage is 68.20513% with 62 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.94%. Comparing base (cec71d6) to head (1cf720c).
⚠️ Report is 2 commits behind head on master.

Files with missing lines Patch % Lines
packages/cubejs-backend-shared/src/env.ts 68.38% 43 Missing ⚠️
...y-orchestrator/src/orchestrator/PreAggregations.ts 40.00% 6 Missing ⚠️
...s/cubejs-clickhouse-driver/src/ClickHouseDriver.ts 63.63% 0 Missing and 4 partials ⚠️
packages/cubejs-druid-driver/src/DruidDriver.ts 20.00% 3 Missing and 1 partial ⚠️
packages/cubejs-base-driver/src/BaseDriver.ts 0.00% 2 Missing ⚠️
...kages/cubejs-bigquery-driver/src/BigQueryDriver.ts 33.33% 0 Missing and 2 partials ⚠️
packages/cubejs-server-core/src/core/server.ts 94.11% 0 Missing and 1 partial ⚠️

❗ There is a different number of reports uploaded between BASE (cec71d6) and HEAD (1cf720c). Click for more details.

HEAD has 1 upload less than BASE
Flag BASE (cec71d6) HEAD (1cf720c)
cubesql 1 0
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #10587       +/-   ##
===========================================
- Coverage   83.40%   57.94%   -25.47%     
===========================================
  Files         250      215       -35     
  Lines       75261    16639    -58622     
  Branches        0     3345     +3345     
===========================================
- Hits        62775     9641    -53134     
+ Misses      12486     6509     -5977     
- Partials        0      489      +489     
Flag Coverage Δ
cube-backend 57.94% <68.20%> (?)
cubesql ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ovr ovr force-pushed the feat/separate-pre-aggregations-driver branch from 35f1f5a to 7d19fce Compare April 10, 2026 15:10
@ovr ovr merged commit 04eed5d into master Apr 10, 2026
103 of 107 checks passed
@ovr ovr deleted the feat/separate-pre-aggregations-driver branch April 10, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cubejs-jdbc-driver data source driver driver:athena Issues related to the AWS Athena driver driver:bigquery Issues related to the BigQuery driver driver:clickhouse Issues related to the ClickHouse driver driver:databricks driver:dremio driver:druid Issues relating to the Druid driver driver:duckdb driver:elasticsearch Issues relating to the ElasticSearch driver driver:firebolt driver:hive Issues relating to the Apache Hive driver driver:mongodb Issues relating to the MongoBI driver driver:mssql Issues relating to the MSSQL driver driver:mysql Issues relating to the MySQL/MariaDB driver driver:mysql-aurora-serverless Issues relating to the AWS MySQL Aurora Serverless driver driver:oracle Issues relating to the Oracle driver driver:pinot driver:postgres Issues relating to the Postgres driver driver:prestodb Issues relating to the PrestoDB driver driver:questdb driver:redshift Issues relating to the Redshift driver driver:snowflake Issues relating to the Snowflake driver driver:sqlite Issues relating to the SQLite driver javascript Pull requests that update Javascript code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants