set default GCS_CLIENT_POOL_SIZE as max(upload_concurrency, download_…

…concurrency) * 3 after merge #753, thanks @minguyen-jumptrading again
Altinity · Oct 10, 2023 · 5c44571 · 5c44571
1 parent d0576f8
commit 5c44571
Show file tree

Hide file tree

Showing 3 changed files with 4 additions and 3 deletions.
diff --git a/ChangeLog.md b/ChangeLog.md
@@ -6,7 +6,8 @@ BUG FIXES
 - fix restore for object disk frozen_metadata.txt fix [752](https://github.com/Altinity/clickhouse-backup/issues/752)
 - fix more corner cases for `check_parts_columns: true`, fix [747](https://github.com/Altinity/clickhouse-backup/issues/747)
 - fix applying macros to s3 endpoint in object disk during restore embedded backups, fix [750](https://github.com/Altinity/clickhouse-backup/issues/750)
-
+- rewrite GCS clients pool, set default GCS_CLIENT_POOL_SIZE as max(upload_concurrency, download_concurrency) * 3  to avoid stuck, fix [753](https://github.com/Altinity/clickhouse-backup/pull/753), thanks @minguyen-jumptrading
+
 # v2.4.1
 IMPROVEMENTS
 - switch to go-1.21

diff --git a/ReadMe.md b/ReadMe.md
@@ -501,7 +501,7 @@ gcs:
   compression_level: 1         # GCS_COMPRESSION_LEVEL
   compression_format: tar      # GCS_COMPRESSION_FORMAT, allowed values tar, lz4, bzip2, gzip, sz, xz, brortli, zstd, `none` for upload data part folders as is
   storage_class: STANDARD      # GCS_STORAGE_CLASS
-  client_pool_size: 500        # GCS_CLIENT_POOL_SIZE, should be at least 2 times bigger than `UPLOAD_CONCURRENCY` or `DOWNLOAD_CONCURRENCY` in each upload and download case
+  client_pool_size: 500        # GCS_CLIENT_POOL_SIZE, default max(upload_concurrency, download concurrency) * 3, should be at least 3 times bigger than `UPLOAD_CONCURRENCY` or `DOWNLOAD_CONCURRENCY` in each upload and download case to avoid stuck
   # GCS_OBJECT_LABELS, allow setup metadata for each object during upload, use {macro_name} from system.macros and {backupName} for current backup name
   # The format for this env variable is "key1:value1,key2:value2". For YAML please continue using map syntax
   object_labels: {}

diff --git a/pkg/config/config.go b/pkg/config/config.go
@@ -571,7 +571,7 @@ func DefaultConfig() *Config {
 			CompressionLevel:  1,
 			CompressionFormat: "tar",
 			StorageClass:      "STANDARD",
-			ClientPoolSize:    500,
+			ClientPoolSize:    int(max(uploadConcurrency, downloadConcurrency)) * 3,
 		},
 		COS: COSConfig{
 			RowURL:            "",