Writes to Snowflake are happening every 60 seconds approx. despite increasing buffer time and checkpointing interval #44
Description
Hi Team,
I am using the SF connector for sinking records from S3. My flink application has a checkpointing interval of 5 minutes. As per the flink documentation, below is what I see about bufferTimeMillis
bufferTimeMillis is the time in milliseconds to buffer records before writing them to Snowflake. NOTE: The buffer is also flushed when a checkpoint is triggered or the job is stopped.
I set the buffer time to a very large value to enable the connector to perform flush only during checkpointing as below
SnowflakeSink.<Event>builder()
.url(snowflakePrope.get("SNOWFLAKE_ACCOUNT"))
.bufferTimeMillis(2678400000L)
.build("Snowflake Sink")
With this, I expected the flush to happen only every 5 mins. But, I noticed the writes to Snowflake table approx. every 60 seconds. The delta stream page about this connector also tells only about checkpointing and buffertime when it comes to flushing as below
In this example, the checkpointing interval is set to 100 milliseconds, and the buffering interval is configured as 1 second. This tells the Flink job to flush the records at least every 100 milliseconds, i.e., on every checkpoint.
Could you please let me know the reason behind this behavior?
Thanks