[SNOW-2464956] Close client on primary deployment change #1078

sfc-gh-aminyaylov · 2025-10-14T02:16:38Z

Problem

When the primary deployment URL is updated, the client throws a CHANNEL_INVALID error. The user responds by re-opening the channel, which succeeds. However, ingestion continues to fail due to the deployment mismatch.

Solution

Client is automatically closed if it detects the primary deployment has changed, forcing the user to recreate the client.

The following errors indicate a primary deployment reconfiguration and will immediately trigger a client close:

Client-side deployment ID mismatch: we are uploading to a different bucket location. The bucket location change occurs during periodic storage credential refresh, which calls Snowflake using the current URL.
Server-side deployment ID mismatch: we are registering a file with the wrong deployment
Server-side encryption key mismatch: we are registering a file with an invalid encryption key

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java

sfc-gh-aminyaylov · 2025-10-22T01:16:21Z

[SNOW-2464956] Close client on primary deployment change #1078 👈 (View in Graphite)
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

sfc-gh-psaha

I am onboard with the high level approach - I'm delegating to @sfc-gh-hmadan for the review.

src/main/java/net/snowflake/ingest/utils/SFException.java

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java

sfc-gh-hmadan · 2025-10-24T05:31:38Z

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java

                                .forEach(
                                    channelStatus -> {
                                      if (channelStatus.getStatusCode() != RESPONSE_SUCCESS) {
+                                        if (isTerminalError(channelStatus.getStatusCode())) {


we are expecting that the service response will still be a valid HTTP 200 response, on which response.getBlobsStatus can successfully fire.
In this situation of failover, I'd expect the service to return. non-200 response (at line 626 : snowflakeServiceClient.registerBlob(request, executionCount)).
Why would we make the service return a HTTP 200, with the inner per-chunk details inside the response payload have this terminal error code?

This turned out to be a rabbit hole. The DEPLOYMENT_ID_MISMATCH is determined while processing chunks. There are many chunks per request. Therefore, these errors accumulate into a 200 OK response object. Meanwhile, the INVALID_ENCRYPTION_KEY error is checked immediately and thrown as a 400 BAD REQUEST. So I split them up and am now handling them separately. PTAL.

sfc-gh-hmadan · 2025-10-24T05:32:06Z

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java

+   * @param statusCode the server response status code
+   * @return true if terminal error, false otherwise
+   */
+  private static boolean isTerminalError(long statusCode) {


nit: if we remove the private qualifier can RegisterService also call this method?

They're different signatures. The errors coming from deeper in the SDK stack (like staging location mismatch) are thrown as SFExceptions. Meanwhile, this method is dealing with response codes from the server JSON response object.

sfc-gh-hmadan · 2025-10-24T05:35:33Z

src/test/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientTest.java

+                + "}",
+            dbName, schemaName, tableName, channelName, channelSequencer);
+
+    apiOverride.addSerializedJsonOverride(


Lets add a true E2E for validating this change works as expected, without any mocks in the picture?
Please check with Alec on how to do so. SDK E2E will only work against prod, by the way.

sfc-gh-cqu · 2025-10-27T12:00:15Z

Thanks for working on this @sfc-gh-aminyaylov! I didn't go through the actual changes, but from the description it might affect the preprod replication testing which has a test case for client redirect. Could you help check if any update to the test is needed when releasing the new SDK? Thanks in advance! https://dp-telemetry-and-streaming-ingest-001.jenkinsdev1.us-west-2.aws-dev.app.snowflake.com/job/SSV1ReplicationTestClientRedirect/
https://github.com/snowflakedb/snowflake/blob/599c7a27370fca42604e1587dffdf8cf5e332678/Tests/system_tests/snowpipe_streaming_replication_test/ssv1_replication_test/tests/ssv1_replication_client_redirect_test.py

sfc-gh-aminyaylov · 2025-10-30T00:59:17Z

Thanks for working on this @sfc-gh-aminyaylov! I didn't go through the actual changes, but from the description it might affect the preprod replication testing which has a test case for client redirect. Could you help check if any update to the test is needed when releasing the new SDK? Thanks in advance! https://dp-telemetry-and-streaming-ingest-001.jenkinsdev1.us-west-2.aws-dev.app.snowflake.com/job/SSV1ReplicationTestClientRedirect/
https://github.com/snowflakedb/snowflake/blob/599c7a27370fca42604e1587dffdf8cf5e332678/Tests/system_tests/snowpipe_streaming_replication_test/ssv1_replication_test/tests/ssv1_replication_client_redirect_test.py

Yeah, good point. The flow is identical except for step 6, where we close the client instead of invalidating the channel. The customer will get a CLOSED_CLIENT error if making any additional calls before recreating.

I'll update the tests.

sfc-gh-hmadan · 2025-11-04T07:45:38Z

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java

+        && ire.getErrorBody().getCode() != null
+        && ire.getErrorBody()
+            .getCode()
+            .equals(


do we need to do a case-insensitive string comparison? Or is every other place in the SDK doing a case sensitive check anyway?

sfc-gh-aminyaylov force-pushed the aminyaylov-client-reconfigure branch from f3c1432 to 0e9fbbf Compare October 14, 2025 02:17

sfc-gh-hmadan reviewed Oct 16, 2025

View reviewed changes

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java Outdated Show resolved Hide resolved

Close client on primary deployment change

186b94a

sfc-gh-aminyaylov force-pushed the aminyaylov-client-reconfigure branch from 0e9fbbf to 186b94a Compare October 22, 2025 01:16

sfc-gh-aminyaylov changed the title ~~Draft client reconfigure mechanism on client redirect~~ Close client on Oct 22, 2025

sfc-gh-aminyaylov changed the title ~~Close client on~~ [SNOW-2193898] Close client on primary deployment change Oct 22, 2025

sfc-gh-aminyaylov requested a review from sfc-gh-psaha October 22, 2025 01:39

sfc-gh-aminyaylov marked this pull request as ready for review October 22, 2025 01:40

sfc-gh-aminyaylov requested review from a team and sfc-gh-tzhang as code owners October 22, 2025 01:40

Add unit tests

4c4c918

sfc-gh-aminyaylov force-pushed the aminyaylov-client-reconfigure branch from a26d901 to 4c4c918 Compare October 22, 2025 01:41

sfc-gh-psaha reviewed Oct 22, 2025

View reviewed changes

Refactor SFException extraction method

969ecff

sfc-gh-aminyaylov requested a review from sfc-gh-hmadan October 24, 2025 00:05

sfc-gh-hmadan reviewed Oct 24, 2025

View reviewed changes

src/main/java/net/snowflake/ingest/utils/SFException.java Show resolved Hide resolved

sfc-gh-hmadan reviewed Oct 24, 2025

View reviewed changes

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java Outdated Show resolved Hide resolved

sfc-gh-hmadan reviewed Oct 24, 2025

View reviewed changes

sfc-gh-aminyaylov changed the title ~~[SNOW-2193898] Close client on primary deployment change~~ [SNOW-2464956] Close client on primary deployment change Oct 27, 2025

sfc-gh-aminyaylov added 2 commits October 29, 2025 15:59

Fix error message in test

1a6831d

Split out errors; minimize API surface

0505045

sfc-gh-aminyaylov force-pushed the aminyaylov-client-reconfigure branch from a486b2b to 0505045 Compare October 29, 2025 23:00

sfc-gh-aminyaylov requested review from sfc-gh-hmadan and sfc-gh-psaha October 29, 2025 23:30

Improve exceptions and error messages

36e0f3e

sfc-gh-aminyaylov added 3 commits October 29, 2025 17:59

Java format

228a7ca

Fix test message

a47336c

Format java

5f10168

sfc-gh-hmadan reviewed Nov 4, 2025

View reviewed changes

sfc-gh-hmadan approved these changes Nov 4, 2025

View reviewed changes

Check client closed on insert rows

376a077

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SNOW-2464956] Close client on primary deployment change #1078

[SNOW-2464956] Close client on primary deployment change #1078

sfc-gh-aminyaylov commented Oct 14, 2025 •

edited

Loading

Uh oh!

Uh oh!

sfc-gh-aminyaylov commented Oct 22, 2025 •

edited

Loading

Uh oh!

sfc-gh-psaha left a comment

Uh oh!

Uh oh!

Uh oh!

sfc-gh-hmadan Oct 24, 2025

Uh oh!

sfc-gh-aminyaylov Oct 29, 2025

Uh oh!

sfc-gh-hmadan Oct 24, 2025

Uh oh!

sfc-gh-aminyaylov Oct 29, 2025

Uh oh!

sfc-gh-hmadan Oct 24, 2025

Uh oh!

sfc-gh-cqu commented Oct 27, 2025

Uh oh!

sfc-gh-aminyaylov commented Oct 30, 2025

Uh oh!

sfc-gh-hmadan Nov 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[SNOW-2464956] Close client on primary deployment change #1078

Are you sure you want to change the base?

[SNOW-2464956] Close client on primary deployment change #1078

Conversation

sfc-gh-aminyaylov commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Uh oh!

Uh oh!

sfc-gh-aminyaylov commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sfc-gh-psaha left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sfc-gh-hmadan Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-aminyaylov Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-hmadan Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-aminyaylov Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-hmadan Oct 24, 2025

Choose a reason for hiding this comment

Uh oh!

sfc-gh-cqu commented Oct 27, 2025

Uh oh!

sfc-gh-aminyaylov commented Oct 30, 2025

Uh oh!

sfc-gh-hmadan Nov 4, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sfc-gh-aminyaylov commented Oct 14, 2025 •

edited

Loading

sfc-gh-aminyaylov commented Oct 22, 2025 •

edited

Loading