Skip to content

feat(eap-items): drop attributes_array JSON column from distributed tables - #8236

Open
onewland wants to merge 2 commits into
masterfrom
oliver/drop-attributes-array-json-dist-only
Open

feat(eap-items): drop attributes_array JSON column from distributed tables#8236
onewland wants to merge 2 commits into
masterfrom
oliver/drop-attributes-array-json-dist-only

Conversation

@onewland

Copy link
Copy Markdown
Contributor

Reprise of #8231, which was reverted (451f9eb): dropping attributes_array from the downsample local MergeTree tables triggered a full part rewrite even though the column was empty, which is unacceptable in customer-facing environments.

This version (migration 0062_drop_attributes_array_json_from_dist) drops the attributes_array JSON column (added in 0050, superseded by the typed attributes_array_{string,int,float,bool} map columns from 0059) from distributed tables only — a metadata-only change with no data rewrites:

  • eap_items_1_dist + downsample _dist tables (EVENTS_ANALYTICS_PLATFORM)
  • eap_items_1_dist_ro + downsample _dist_ro tables (EVENTS_ANALYTICS_PLATFORM_RO, inherited via CREATE TABLE ... AS in 0056)

Notes:

  • All local (MergeTree) tables keep the column. The downsample materialized views never projected attributes_array (verified across all MV versions mv_1–mv_9), so the column holds only default values there and nothing references it.
  • No view regeneration needed. Backwards ops re-add the column with its original 0050 definition.
  • Consumer stopped writing the column in Stop writing to the attributes_array column in the EAP consumer #8179 (2026-07-14); with ~90d retention, the last rows that could contain data in eap_items_1_local age out around mid-October 2026, after which the local-table column could be revisited.

@github-actions

Copy link
Copy Markdown

This PR has a migration; here is the generated SQL for ./snuba/migrations/groups.py ()

-- start migrations

-- forward migration events_analytics_platform : 0062_drop_attributes_array_json_from_dist
Distributed op: ALTER TABLE eap_items_1_dist ON CLUSTER 'cluster_one_sh' DROP COLUMN IF EXISTS attributes_array;
Distributed op: ALTER TABLE eap_items_1_downsample_8_dist ON CLUSTER 'cluster_one_sh' DROP COLUMN IF EXISTS attributes_array;
Distributed op: ALTER TABLE eap_items_1_downsample_64_dist ON CLUSTER 'cluster_one_sh' DROP COLUMN IF EXISTS attributes_array;
Distributed op: ALTER TABLE eap_items_1_downsample_512_dist ON CLUSTER 'cluster_one_sh' DROP COLUMN IF EXISTS attributes_array;
Distributed op: ALTER TABLE eap_items_1_dist_ro ON CLUSTER 'cluster_one_sh' DROP COLUMN IF EXISTS attributes_array;
Distributed op: ALTER TABLE eap_items_1_downsample_8_dist_ro ON CLUSTER 'cluster_one_sh' DROP COLUMN IF EXISTS attributes_array;
Distributed op: ALTER TABLE eap_items_1_downsample_64_dist_ro ON CLUSTER 'cluster_one_sh' DROP COLUMN IF EXISTS attributes_array;
Distributed op: ALTER TABLE eap_items_1_downsample_512_dist_ro ON CLUSTER 'cluster_one_sh' DROP COLUMN IF EXISTS attributes_array;
-- end forward migration events_analytics_platform : 0062_drop_attributes_array_json_from_dist




-- backward migration events_analytics_platform : 0062_drop_attributes_array_json_from_dist
Distributed op: ALTER TABLE eap_items_1_downsample_512_dist_ro ON CLUSTER 'cluster_one_sh' ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Distributed op: ALTER TABLE eap_items_1_downsample_64_dist_ro ON CLUSTER 'cluster_one_sh' ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Distributed op: ALTER TABLE eap_items_1_downsample_8_dist_ro ON CLUSTER 'cluster_one_sh' ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Distributed op: ALTER TABLE eap_items_1_dist_ro ON CLUSTER 'cluster_one_sh' ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Distributed op: ALTER TABLE eap_items_1_downsample_512_dist ON CLUSTER 'cluster_one_sh' ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Distributed op: ALTER TABLE eap_items_1_downsample_64_dist ON CLUSTER 'cluster_one_sh' ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Distributed op: ALTER TABLE eap_items_1_downsample_8_dist ON CLUSTER 'cluster_one_sh' ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
Distributed op: ALTER TABLE eap_items_1_dist ON CLUSTER 'cluster_one_sh' ADD COLUMN IF NOT EXISTS attributes_array JSON(max_dynamic_paths=128) CODEC (ZSTD(1)) AFTER attributes_float_39;
-- end backward migration events_analytics_platform : 0062_drop_attributes_array_json_from_dist

@onewland
onewland marked this pull request as ready for review July 29, 2026 15:40
@onewland
onewland requested review from a team as code owners July 29, 2026 15:40
@onewland onewland changed the title feat(eap-items): drop attributes_array JSON column from distributed tables only feat(eap-items): drop attributes_array JSON column from distributed tables Jul 29, 2026
Comment on lines +60 to +69
return [
operations.AddColumn(
storage_set=ss,
table_name=table,
column=attributes_array_column,
after=after,
target=OperationTarget.DISTRIBUTED,
)
for (ss, table) in reversed(_tables())
]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not add the column back in case we need to go backwards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants