Infinite option for bigQueryRetry

[aakarim]

Hi, thanks for the connector!

I have a fairly low daily limit on the BigQuery Query Usage per day quota to control costs, which we fairly regularly hit causing the connector to stop running and enter a failed state. We have the option to set bigQueryRetry but it's capped, setting 0 results in no retries. We then have to manually reset the connector every day when the quota has refreshed.

Would it be possible to allow infinite retries so the connector can just pick up the next day?

[b-goyal]

@aakarim, what do you mean by it's capped . I think only condition is minimum 0. If you set it to a large value, doesn't it retry?
Could you check if setting bigQueryRetryWait and bigQueryRetry both to large values help ?

[aakarim]

@b-goyal Yes it does retry, it just only retries set number of times. I'd like it to keep retrying until the quota has expired (usually after a day). I could multiply it out, so if I have a bigQueryRetryWait of 15s that means that I need to retry 5,760 times max before giving up, but I'd like the flexibility to just set it as infinite and have it keep retrying forever.

[b-goyal]

bigQueryRetry is of type int so it is basically capped at 2147483647.
The bigQueryRetryWait is defaulted to 1000ms ( 1sec) . Doc link : https://docs.confluent.io/kafka-connectors/bigquery/current/kafka_connect_bigquery_config.html#:~:text=Importance%3A%20medium-,bigQueryRetryWait,-The%20minimum%20amount
If you set bigQueryRetry to the highest value possible which is 2147483647, with 1sec wait before retry, you will be retrying for 68 years.
I believe we do not need any explicit value to make the connector retry for days, if we set it to max, it should serve the purpose.
Does this help?