Skip to content

Commit 9f2f02d

Browse files
author
Nikita Unisikhin
committed
Merge pull request #1348 in CLOUD/terraform-provider-yandex-mirror from mdb-clickhouse-new-settings to master
Squashed commit of the following: commit 3e3bae2a58c5aa67825f7b8753844421738fc908 Author: Nikita Unisikhin <[email protected]> Date: Wed Aug 28 03:02:03 2024 +0300 Fix new bool fields commit b9ab6afb2a2e0d6f602286abb5c695800d9c42f8 Author: Nikita Unisikhin <[email protected]> Date: Mon Aug 26 15:55:41 2024 +0300 Update CHANGELOG commit 3ea4d2bd943b6e3f1f365f95b707cdd5c1cafd03 Author: Nikita Unisikhin <[email protected]> Date: Thu Aug 22 10:26:28 2024 +0300 Added new clickhouse settings
1 parent 8acaada commit 9f2f02d

14 files changed

+574
-10
lines changed
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
kind: FEATURES
2+
body: 'clickhouse: added `log_query_threads`, `max_insert_threads`, `use_hedged_requests`, `idle_connection_timeout` for user settings'
3+
time: 2024-08-26T15:54:49.510442+03:00
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
kind: FEATURES
2+
body: 'clickhouse: added `hedged_connection_timeout_ms`, `load_balancing`, `prefer_localhost_replica` for user settings'
3+
time: 2024-08-26T15:54:55.737175+03:00
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
kind: FEATURES
2+
body: 'clickhouse: added `merge_max_block_size`, `check_sample_column_is_correct` for merge tree configuration'
3+
time: 2024-08-26T15:55:03.102777+03:00
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
kind: FEATURES
2+
body: 'clickhouse: added `max_merge_selecting_sleep_ms`, `max_cleanup_delay_period` for merge tree configuration'
3+
time: 2024-08-26T15:55:09.707815+03:00
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
kind: FEATURES
2+
body: 'clickhouse: added `path_column_name`, `time_column_name`, `value_column_name`, `version_column_name` for graphite configuration'
3+
time: 2024-08-26T15:55:17.1473+03:00
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
kind: FEATURES
2+
body: 'clickhouse: added `debug`, `auto_offset_reset` for kafka integration settings'
3+
time: 2024-08-26T15:55:22.324701+03:00
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
kind: FEATURES
2+
body: 'clickhouse: added `query_masking_rules`, `dictionaries_lazy_load`, `query_cache` for server configuration'
3+
time: 2024-08-26T15:55:28.715633+03:00

go.mod

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ require (
3636
github.com/spf13/cobra v1.8.0
3737
github.com/stretchr/objx v0.5.0
3838
github.com/stretchr/testify v1.8.4
39-
github.com/yandex-cloud/go-genproto v0.0.0-20240805150959-00c80a383bd3
39+
github.com/yandex-cloud/go-genproto v0.0.0-20240819112322-98a264d392f6
4040
github.com/yandex-cloud/go-sdk v0.0.0-20240805151354-29f07dbe484e
4141
github.com/ydb-platform/terraform-provider-ydb v0.0.21
4242
golang.org/x/crypto v0.21.0

go.sum

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -770,8 +770,9 @@ github.com/xen0n/gosmopolitan v1.2.1 h1:3pttnTuFumELBRSh+KQs1zcz4fN6Zy7aB0xlnQSn
770770
github.com/xen0n/gosmopolitan v1.2.1/go.mod h1:JsHq/Brs1o050OOdmzHeOr0N7OtlnKRAGAsElF8xBQA=
771771
github.com/yagipy/maintidx v1.0.0 h1:h5NvIsCz+nRDapQ0exNv4aJ0yXSI0420omVANTv3GJM=
772772
github.com/yagipy/maintidx v1.0.0/go.mod h1:0qNf/I/CCZXSMhsRsrEPDZ+DkekpKLXAJfsTACwgXLk=
773-
github.com/yandex-cloud/go-genproto v0.0.0-20240805150959-00c80a383bd3 h1:5cPFwSkj7HFyFwystyt4UXvAulxdjWAOMXpY3OQH9hk=
774773
github.com/yandex-cloud/go-genproto v0.0.0-20240805150959-00c80a383bd3/go.mod h1:HEUYX/p8966tMUHHT+TsS0hF/Ca/NYwqprC5WXSDMfE=
774+
github.com/yandex-cloud/go-genproto v0.0.0-20240819112322-98a264d392f6 h1:w57l27dDkJTVSi8hM3H/WVkiv+CsJwAIweqO6pFdljk=
775+
github.com/yandex-cloud/go-genproto v0.0.0-20240819112322-98a264d392f6/go.mod h1:HEUYX/p8966tMUHHT+TsS0hF/Ca/NYwqprC5WXSDMfE=
775776
github.com/yandex-cloud/go-sdk v0.0.0-20240805151354-29f07dbe484e h1:eprYcOC0KCmaFNfUQNrovFboWq6dLFWYm20M22uUs5Q=
776777
github.com/yandex-cloud/go-sdk v0.0.0-20240805151354-29f07dbe484e/go.mod h1:7OD14iLpLhBeaRZokSeNRvp58RIAEEmc7fj5z9SjzAA=
777778
github.com/ydb-platform/terraform-provider-ydb v0.0.21 h1:YWiZG/2tquBuVBLlgyhwnpaHwLUGzEMM0unfG1h0Z88=

website/docs/d/datasource_mdb_clickhouse_cluster.html.markdown

Lines changed: 33 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -244,6 +244,13 @@ The `settings` block supports:
244244
* `memory_overcommit_ratio_denominator` - (Optional) It represents soft memory limit in case when hard limit is reached on user level. This value is used to compute overcommit ratio for the query. Zero means skip the query.
245245
* `memory_overcommit_ratio_denominator_for_user` - (Optional) It represents soft memory limit in case when hard limit is reached on global level. This value is used to compute overcommit ratio for the query. Zero means skip the query.
246246
* `memory_usage_overcommit_max_wait_microseconds` - (Optional) Maximum time thread will wait for memory to be freed in the case of memory overcommit on a user level. If the timeout is reached and memory is not freed, an exception is thrown.
247+
* `log_query_threads` - (Optional) Setting up query threads logging. Query threads log into the system.query_thread_log table. This setting has effect only when log_queries is true. Queries’ threads run by ClickHouse with this setup are logged according to the rules in the query_thread_log server configuration parameter. Default value: true.
248+
* `max_insert_threads` - (Optional) The maximum number of threads to execute the INSERT SELECT query. Default value: 0.
249+
* `use_hedged_requests` - (Optional) Enables hedged requests logic for remote queries. It allows to establish many connections with different replicas for query. New connection is enabled in case existent connection(s) with replica(s) were not established within hedged_connection_timeout or no data was received within receive_data_timeout. Query uses the first connection which send non empty progress packet (or data packet, if allow_changing_replica_until_first_data_packet); other connections are cancelled. Queries with max_parallel_replicas > 1 are supported. Default value: true.
250+
* `idle_connection_timeout` - (Optional) Timeout to close idle TCP connections after specified number of seconds. Default value: 3600 seconds.
251+
* `hedged_connection_timeout_ms` - (Optional) Connection timeout for establishing connection with replica for Hedged requests. Default value: 50 milliseconds.
252+
* `load_balancing` - (Optional) Specifies the algorithm of replicas selection that is used for distributed query processing, one of: random, nearest_hostname, in_order, first_or_random, round_robin. Default value: random.
253+
* `prefer_localhost_replica` - (Optional) Enables/disables preferable using the localhost replica when processing distributed queries. Default value: true.
247254

248255
The `quota` block supports:
249256

@@ -320,7 +327,7 @@ The `config` block supports:
320327
`part_log_retention_size`, `part_log_retention_time`, `metric_log_enabled`, `metric_log_retention_size`, `metric_log_retention_time`,
321328
`trace_log_enabled`, `trace_log_retention_size`, `trace_log_retention_time`, `text_log_enabled`, `text_log_retention_size`,
322329
`text_log_retention_time`, `text_log_level`, `background_pool_size`, `background_schedule_pool_size`, `background_fetches_pool_size`, `background_message_broker_schedule_pool_size`, `background_merges_mutations_concurrency_ratio`, `default_database`,
323-
`total_memory_profiler_step` - ClickHouse server parameters. For more information, see
330+
`total_memory_profiler_step`, `dictionaries_lazy_load` - ClickHouse server parameters. For more information, see
324331
[the official documentation](https://cloud.yandex.com/docs/managed-clickhouse/concepts/settings-list).
325332

326333
* `merge_tree` - MergeTree engine configuration. The structure is documented below.
@@ -329,6 +336,8 @@ The `config` block supports:
329336
* `compression` - Data compression configuration. The structure is documented below.
330337
* `rabbitmq` - RabbitMQ connection configuration. The structure is documented below.
331338
* `graphite_rollup` - Graphite rollup configuration. The structure is documented below.
339+
* `query_masking_rules` - Query masking rules configuration. The structure is documented below.
340+
* `query_cache` - Query cache configuration. The structure is documented below.
332341

333342
The `merge_tree` block supports:
334343

@@ -352,6 +361,10 @@ The `merge_tree` block supports:
352361
* `min_age_to_force_merge_seconds` - (Optional) Merge parts if every part in the range is older than the value of `min_age_to_force_merge_seconds`.
353362
* `min_age_to_force_merge_on_partition_only` - (Optional) Whether min_age_to_force_merge_seconds should be applied only on the entire partition and not on subset.
354363
* `merge_selecting_sleep_ms` - (Optional) Sleep time for merge selecting when no part is selected. A lower setting triggers selecting tasks in background_schedule_pool frequently, which results in a large number of requests to ClickHouse Keeper in large-scale clusters.
364+
* `merge_max_block_size` - (Optional) The number of rows that are read from the merged parts into memory. Default value: 8192.
365+
* `check_sample_column_is_correct` - (Optional) Enables the check at table creation, that the data type of a column for sampling or sampling expression is correct. The data type must be one of unsigned integer types: UInt8, UInt16, UInt32, UInt64. Default value: true.
366+
* `max_merge_selecting_sleep_ms` - (Optional) Maximum sleep time for merge selecting, a lower setting will trigger selecting tasks in background_schedule_pool frequently which result in large amount of requests to zookeeper in large-scale clusters. Default value: 60000 milliseconds (60 seconds).
367+
* `max_cleanup_delay_period` - (Optional) Maximum period to clean old queue logs, blocks hashes and parts. Default value: 300 seconds.
355368

356369
The `kafka` block supports:
357370

@@ -362,6 +375,8 @@ The `kafka` block supports:
362375
* `enable_ssl_certificate_verification` - (Optional) enable verification of SSL certificates.
363376
* `max_poll_interval_ms` - (Optional) Maximum allowed time between calls to consume messages (e.g., rd_kafka_consumer_poll()) for high-level consumers. If this interval is exceeded the consumer is considered failed and the group will rebalance in order to reassign the partitions to another consumer group member.
364377
* `session_timeout_ms` - (Optional) Client group session and failure detection timeout. The consumer sends periodic heartbeats (heartbeat.interval.ms) to indicate its liveness to the broker. If no hearts are received by the broker for a group member within the session timeout, the broker will remove the consumer from the group and trigger a rebalance.
378+
* `debug` - (Optional) A comma-separated list of debug contexts to enable.
379+
* `auto_offset_reset` - (Optional) Action to take when there is no initial offset in offset store or the desired offset is out of range: 'smallest','earliest' - automatically reset the offset to the smallest offset, 'largest','latest' - automatically reset the offset to the largest offset, 'error' - trigger an error (ERR__AUTO_OFFSET_RESET) which is retrieved by consuming messages and checking 'message->err'.
365380

366381
The `kafka_topic` block supports:
367382

@@ -390,6 +405,23 @@ The `graphite_rollup` block supports:
390405
* `retention` - Retain parameters.
391406
* `age` - Minimum data age in seconds.
392407
* `precision` - Accuracy of determining the age of the data in seconds.
408+
* `path_column_name` - (Optional) The name of the column storing the metric name (Graphite sensor). Default value: Path.
409+
* `time_column_name` - (Optional) The name of the column storing the time of measuring the metric. Default value: Time.
410+
* `value_column_name` - (Optional) The name of the column storing the value of the metric at the time set in time_column_name. Default value: Value.
411+
* `version_column_name` - (Optional) The name of the column storing the version of the metric. Default value: Timestamp.
412+
413+
The `query_masking_rules` block supports:
414+
415+
* `name` - (Optional) Name for the rule.
416+
* `regexp` - (Required) RE2 compatible regular expression.
417+
* `replace` - (Optional) Substitution string for sensitive data. Default value: six asterisks.
418+
419+
The `query_cache` block supports:
420+
421+
* `max_size_in_bytes` - (Optional) The maximum cache size in bytes. 0 means the query cache is disabled. Default value: 1073741824 (1 GiB).
422+
* `max_entries` - (Optional) The maximum number of SELECT query results stored in the cache. Default value: 1024.
423+
* `max_entry_size_in_bytes` - (Optional) The maximum size in bytes SELECT query results may have to be saved in the cache. Default value: 1048576 (1 MiB).
424+
* `max_entry_size_in_rows` - (Optional) The maximum number of rows SELECT query results may have to be saved in the cache. Default value: 30000000 (30 mil).
393425

394426
The `cloud_storage` block supports:
395427

0 commit comments

Comments
 (0)