Skip to content

Specify explicit field masks on regional SQL Connect Get/List calls - #11536

Open
lfkellogg wants to merge 7 commits into
mainfrom
lkellogg/fdc-field-masks
Open

lfkellogg wants to merge 7 commits into
mainfrom
lkellogg/fdc-field-masks

Conversation

@lfkellogg

@lfkellogg lfkellogg commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

NOTE: This CL replaces #11321 (identical commits). Re-opened from an in-repo branch because the CI workflow now only runs integration tests for PRs whose head branch lives in firebase/firebase-tools; fork-based PRs leave the required integration checks permanently "expected".

Context: b/534865211

Specifying precise field masks will ensure we only request the fields we actually need, allowing the backend to optimize resource usage and compute expensive fields on-demand.

Note: this change excludes cross-location aggregate list requests (.../locations/-/...), which do not yet support field masks (b/442666798).

Scenarios Tested (paste into PR)

  • dataconnect:services:list — per-service GetSchema sends fields=datasources,updateTime, ListConnectors sends fields=connectors.name,connectors.updateTime,nextPageToken. Table output (instance, database, schema/connector update times) matches unmasked curl of the same resources.
  • deploy --only dataconnect --dry-run — ensureServiceIsConnectedToCloudSql fetches the schema with no fields param (full object, all 8 keys) before the validateOnly upsert. Confirms both the full-fetch-before-upsert fix and that an empty mask omits the parameter entirely.
  • deploy --only dataconnect (new service, schema + 2 connectors) — end-to-end create: service LRO, Cloud SQL link, SQL migration, connectors. Release phase issues ListConnectors/ListSchemas with fields=<name>,nextPageToken; deploy completes.
  • deploy --only dataconnect after removing one connector locally — masked ListConnectors (name-only) still identifies the orphaned second connector; DELETE …/connectors/second is issued and succeeds.
  • nextPageToken masking behaviour (curl, ListServices?pageSize=1) — mask services.name → nextPageToken stripped; services.name,nextPageToken → present; fields= (empty) → full response.
  • Unit: client.spec.ts, release.spec.ts (nock URLs assert the masks), schemaMigration.spec.ts, init/dataconnect/index.spec.ts — all passing.

Sample Commands (paste into PR)

# Read-only listing with field masks on GetSchema / ListConnectors
firebase dataconnect:services:list --project <project>

# Full schema fetch before upsert (no field mask), validateOnly upsert
firebase deploy --only dataconnect --dry-run --project <project>

# Release phase: ListSchemas/ListConnectors with name-only masks + nextPageToken,
# including deletion of connectors removed from dataconnect.yaml
firebase deploy --only dataconnect --project <project> --force

@lfkellogg
lfkellogg requested review from fredzqm and joehan November 17, 2026 14:45

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces field masking support for Data Connect service, schema, and connector API calls to improve performance and reduce payload sizes. The changes include updating client methods to accept optional field arrays, modifying command-line tools and deployment logic to utilize these masks, and updating corresponding unit tests. Per the repository style guide, I have flagged several instances where static configuration arrays should be extracted to top-level constants and marked as 'as const' or 'readonly' to avoid unnecessary re-allocation and ensure consistency.

Comment thread src/dataconnect/client.ts Resolved
Comment thread src/dataconnect/client.ts Resolved
Comment thread src/dataconnect/client.ts Resolved
Comment thread src/deploy/dataconnect/release.ts Resolved
Comment thread src/deploy/dataconnect/release.ts Resolved
@fredzqm

fredzqm commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Thank you! Did an audit

  • In PR Specify explicit field masks on regional SQL Connect Get/List calls #11536's own diff: No bugs and no missing or incorrect field masks. Every field that dataconnect-services-list.ts, release.ts, and schemaMigration.ts depend on is preserved, and the masks are minimal (the only optional nit is that getSchema in dataconnect-services-list.ts asks for "datasources" rather than "datasources.postgresql.cloudSql.instance,datasources.postgresql.database", which is negligible).
  • Outside the PR diff (pre-existing bug in src/init/features/dataconnect/index.ts):
    listSchemas (["schemas.name", "schemas.datasources", "schemas.source"]) and listConnectors (["connectors.name", "connectors.source.files"]) both omit "nextPageToken". Because CCFE strips nextPageToken when it is not in fields, pagination in client.listSchemas / client.listConnectors will silently stop after 100 items.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants