[DO NOT MERGE] Validate beam 272 rc5 by Amar3tto · Pull Request #3505 · GoogleCloudPlatform/DataflowTemplates

Amar3tto · 2026-03-16T20:19:21Z

No description provided.

gemini-code-assist · 2026-03-16T20:19:42Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request focuses on updating and validating the project's dependencies against Apache Beam version 2.72 release candidate 2. It involves a comprehensive upgrade of the Beam version in both Java (Maven pom.xml) and Python (various requirements.txt files), alongside necessary adjustments to dependency hashes and the inclusion/removal of specific Google Cloud client libraries. The changes are part of a pre-merge validation process to ensure full compatibility and stability with the upcoming Beam release.

Highlights

Apache Beam Version Update (Java): Updated the Apache Beam version from 2.71.0 to 2.72.0 in the pom.xml file, along with adding a new Beam Maven repository URL.
Maven Profile Activation: Modified the activation condition for the validateCandidate profile in pom.xml.
Apache Beam Version Update (Python): Upgraded the apache-beam dependency to 2.72.0rc2 across python/default_base_python_requirements.txt, python/default_base_yaml_requirements.txt, and python/src/main/python/streaming-llm/base_requirements.txt.
Python Requirements Regeneration: Regenerated the python/src/main/python/streaming-llm/requirements.txt file, updating apache-beam[gcp] to 2.72.0rc2 with new hashes, correcting pip-compile command paths in comments, and adjusting other dependency references.
Python Dependency Changes: Removed google-cloud-pubsublite and overrides dependencies, and added google-cloud-build as a new dependency within python/src/main/python/streaming-llm/requirements.txt.

Changelog

pom.xml
- Updated beam.version and beam-python.version from 2.71.0 to 2.72.0.
- Added a beam-maven-repo URL for the new Beam version.
- Modified the validateCandidate profile activation condition.
python/default_base_python_requirements.txt
- Updated apache-beam[gcp] dependency from 2.71.0 to 2.72.0rc2.
python/default_base_yaml_requirements.txt
- Updated apache-beam[dataframe,gcp,test,yaml] dependency from 2.71.0 to 2.72.0rc2.
python/src/main/python/streaming-llm/base_requirements.txt
- Updated apache-beam[gcp] dependency from 2.71.0 to 2.72.0rc2.
python/src/main/python/streaming-llm/requirements.txt
- Updated apache-beam[gcp] dependency to 2.72.0rc2 and regenerated its associated hashes.
- Corrected the pip-compile command path in the autogenerated header comment.
- Removed google-cloud-pubsublite and overrides dependencies and their hashes.
- Added google-cloud-build dependency and its hashes.
- Adjusted comments for pillow, torch, torchvision, transformers, and setuptools to reflect updated relative paths for their source requirements.

Activity

This pull request is explicitly marked as [DO NOT MERGE], indicating its purpose is for validation of Beam 2.72 rc2.
No explicit review comments or activity have been recorded in the provided context since the pull request was created.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

codecov · 2026-03-16T21:53:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 52.19%. Comparing base (b8c4776) to head (619bb0b).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3505      +/-   ##
============================================
- Coverage     52.19%   52.19%   -0.01%     
+ Complexity     6048     6045       -3     
============================================
  Files          1040     1040              
  Lines         62985    62985              
  Branches       6901     6901              
============================================
- Hits          32877    32874       -3     
- Misses        27882    27888       +6     
+ Partials       2226     2223       -3

Components	Coverage Δ
spanner-templates	`72.23% <ø> (+0.04%)`	⬆️
spanner-import-export	`69.03% <ø> (+0.12%)`	⬆️
spanner-live-forward-migration	`80.49% <ø> (ø)`
spanner-live-reverse-replication	`77.88% <ø> (ø)`
spanner-bulk-migration	`89.19% <ø> (ø)`
gcs-spanner-dv	`85.34% <ø> (ø)`

Files with missing lines	Coverage Δ
...oud/teleport/v2/templates/GoogleAdsToBigQuery.java	`0.00% <ø> (ø)`
...t/v2/transforms/GoogleAdsRowToReportRowJsonFn.java	`54.00% <ø> (ø)`
...port/v2/utils/GoogleAdsRateLimitPolicyFactory.java	`66.66% <ø> (ø)`
...google/cloud/teleport/v2/utils/GoogleAdsUtils.java	`94.11% <ø> (ø)`

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Amar3tto · 2026-03-17T18:43:05Z

Failing tests:

datastream-to-bigquery
kafka-to-bigquery

googlecloud-to-googlecloud (SpannerChangeStreamsToGcsTest)

Abacn · 2026-03-18T01:06:04Z

Good news is Spanner PR passed at once. Previously it was most likely catching breaking changes

Amar3tto · 2026-03-18T14:35:46Z

Failing tests:

datastream-to-bigquery kafka-to-bigquery

googlecloud-to-googlecloud (SpannerChangeStreamsToGcsTest)

@Abacn could you please check bigquery jobs? what is the actual error

Abacn · 2026-03-18T14:37:24Z

@Abacn could you please check bigquery jobs? what is the actual error

ack. checking

Abacn · 2026-03-18T20:06:52Z

Need to bump dataflow v1b3 proto version:

DataflowTemplates/it/pom.xml

Line 34 in 1efd3b9

<dataflow-api.version>v1b3-rev20250519-2.0.0</dataflow-api.version>

to rev20260118. Beam required this after apache/beam#37462

ERROR 2026-03-18T15:28:48.518Z [severity: ERROR] Uncaught exception in main thread. Exiting with status code 1.
ERROR 2026-03-18T15:28:48.519Z [severity: ERROR] java.lang.NoSuchMethodError: 'java.lang.Integer com.google.api.services.dataflow.model.StreamingConfigTask.getStreamingEngineStateTagEncodingVersion()'
ERROR 2026-03-18T15:28:48.519Z [severity: ERROR] at org.apache.beam.runners.dataflow.worker.streaming.config.StreamingEngineComputationConfigFetcher.createPipelineConfig(StreamingEngineComputationConfigFetcher.java:214)
ERROR 2026-03-18T15:28:48.519Z [severity: ERROR] at org.apache.beam.runners.dataflow.worker.streaming.config.StreamingEngineComputationConfigFetcher.lambda$fetchGlobalConfig$6(StreamingEngineComputationConfigFetcher.java:313)
ERROR 2026-03-18T15:28:48.519Z [severity: ERROR] at java.base/java.util.Optional.map(Optional.java:260)
ERROR 2026-03-18T15:28:48.519Z [severity: ERROR] at org.apache.beam.runners.dataflow.worker.streaming.config.StreamingEngineComputationConfigFetcher.fetchGlobalConfig(StreamingEngineComputationConfigFetcher.java:313)
ERROR 2026-03-18T15:28:48.519Z [severity: ERROR] at org.apache.beam.runners.dataflow.worker.streaming.config.StreamingEngineComputationConfigFetcher.fetchInitialPipelineGlobalConfig(StreamingEngineComputationConfigFetcher.java:298)
ERROR 2026-03-18T15:28:48.519Z [severity: ERROR] at org.apache.beam.runners.dataflow.worker.streaming.config.StreamingEngineComputationConfigFetcher.start(StreamingEngineComputationConfigFetcher.java:240)
ERROR 2026-03-18T15:28:48.520Z [severity: ERROR] at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.start(StreamingDataflowWorker.java:1067)
ERROR 2026-03-18T15:28:48.520Z [severity: ERROR] at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.main(StreamingDataflowWorker.java:1010)

Abacn · 2026-03-18T23:07:23Z

python/src/main/python/job-builder-server/requirements.txt

-aiofiles==25.1.0 \
-    --hash=sha256:a8d728f0a29de45dc521f18f07297428d56992a742f0cd2701ba86e44d23d5b2 \
-    --hash=sha256:abe311e527c862958650f9438e859c1fa7568a141b22abcd015e120e86a85695
+aiofiles==24.1.0 \


it downgraded versions. Consider delete **/__build__/ under DataflowTemplates folder to start a fresh virtual environment

Abacn · 2026-03-19T20:31:07Z

It seems we need to rebase the PR onto latest main branch to resolve errors

Amar3tto · 2026-03-20T11:32:43Z

Only googlecloud-to-clickhouse is failing:
BigQueryToClickHouseIT.setup:74 » ContainerLaunch Container startup failed for image clickhouse/clickhouse-server:23.8

Abacn · 2026-03-20T17:01:38Z

Only googlecloud-to-clickhouse is failing: BigQueryToClickHouseIT.setup:74 » ContainerLaunch Container startup failed for image clickhouse/clickhouse-server:23.8

Likely due to apache/beam#37611, we need to sync the change here

https://github.com/apache/beam/blob/c05733a3a4dfdec99af6ad68c1c4187f595b5b43/it/clickhouse/build.gradle#L30

and other places.

I always think putting resouce manager in Beam repo wasn't a great design decision. For now let's fork ClickHouseResourceManager into Template repo, like here: #3207

Abacn · 2026-03-25T15:20:53Z

cc: @BentsiLeviav would you mind taking a look at #3505 (comment)? Clickhouse client upgrade in Beam 2.72.0 likely needs some changes in Dataflow Templates

BentsiLeviav · 2026-03-25T15:46:03Z

Hi folks

Sorry for that, checking

Abacn · 2026-03-25T17:55:32Z

From the log, the testcontainer startup is successful:

[docker-java-stream-1219201400] INFO org.apache.beam.it.testcontainers.TestContainerResourceManager - clickhouse/clickhouse-server:23.8: Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.

[docker-java-stream-1219201400] INFO org.apache.beam.it.testcontainers.TestContainerResourceManager - clickhouse/clickhouse-server:23.8: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/config.xml'.

[docker-java-stream-1219201400] INFO org.apache.beam.it.testcontainers.TestContainerResourceManager - clickhouse/clickhouse-server:23.8: Processing configuration file '/etc/clickhouse-server/users.xml'.

[docker-java-stream-1219201400] INFO org.apache.beam.it.testcontainers.TestContainerResourceManager - clickhouse/clickhouse-server:23.8: Merging configuration file '/etc/clickhouse-server/users.d/default-user.xml'.

[docker-java-stream-1219201400] INFO org.apache.beam.it.testcontainers.TestContainerResourceManager - clickhouse/clickhouse-server:23.8: Saved preprocessed configuration to '/var/lib/clickhouse/preprocessed_configs/users.xml'.

[pool-3-thread-2] ERROR tc.clickhouse/clickhouse-server:23.8 - Could not start container
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching 'null'
	at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47)
	at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
	at org.testcontainers.clickhouse.ClickHouseContainer.waitUntilContainerStarted(ClickHouseContainer.java:144)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:492)

The line "Timed out waiting for log output matching 'null'" is suspicious. May be related to apache/beam@695498f

BentsiLeviav · 2026-03-25T18:03:49Z

The failure occurs due to the upgrade in the test containers version: apache/beam@695498f from version 1.19.7 to 1.21.4.

In test containers version 1.20.4, the wait strategy for ClickHouse changed.

ClickHouseContainer added an explicit waitUntilContainerStarted() override that now calls getWaitStrategy().waitUntilReady(this). Previously, it inherited JdbcDatabaseContainer's implementation, which did a JDBC connectivity check and never touched the configured wait strategy.

ClickHouseResourceManager.buildContainer() overrides the container's built-in HTTP wait strategy with new LogMessageWaitStrategy().withStartupTimeout(...) without ever calling .withRegEx(...), leaving the pattern as null. LogMessageWaitStrategy requires a regex to
know what log line to wait for, and without one, the field defaults to null, which can never match.

This was harmless before 1.20.4 because the configured wait strategy was never invoked. Now it is, producing this Timed out waiting for log output matching 'null'.

BentsiLeviav · 2026-03-25T18:07:17Z

@Abacn sorry, just saw your comment ^_^

I think removing the setWaitStrategy call in ClickHouseResourceManager.buildContainer() might help. ClickHouseContainer already sets the correct HTTP wait strategy (GET / → 200 "Ok."); we're just clobbering it with a broken one.

I can try to fix and test it, but I’m wondering what the expected timeline is for this hotfix?

Abacn · 2026-03-25T18:37:20Z

Thanks. Opened apache/beam#37951. For DataflowTemplates, testing a workaround in #3557

pull-request-size bot added the size/XXL label Mar 16, 2026

Amar3tto added ignore-for-release and removed size/XXL labels Mar 16, 2026

pull-request-size bot added the size/XXL label Mar 17, 2026

Amar3tto force-pushed the validate-beam-272-rc2 branch 2 times, most recently from 4ee36e1 to 0a96b32 Compare March 18, 2026 14:34

Abacn reviewed Mar 18, 2026

View reviewed changes

Amar3tto force-pushed the validate-beam-272-rc2 branch from 3c9be4c to 89c65a0 Compare March 20, 2026 08:34

Amar3tto added 12 commits March 25, 2026 13:18

Update pom and requirements for 2.72.0 RC2

5c2e494

Update google-ads version

d59634f

Update python requirements for 2.72.0 rc2

09712eb

Use v23 google ads

462f2a0

Fix google-ads-to-googlecloud

d4d839e

Fix assert

fc3a09b

Update dataflow api version

934854c

Fix start_date_time

8257035

Update spanner version

5378fef

Update spanner version

66ac0c5

Remove expected exception text

5f6a158

Fix expected exceptions

4fcce6d

Use RC5

619bb0b

Amar3tto force-pushed the validate-beam-272-rc2 branch from c741eba to 619bb0b Compare March 25, 2026 09:18

Amar3tto changed the title ~~[DO NOT MERGE] Validate beam 272 rc2~~ [DO NOT MERGE] Validate beam 272 rc5 Mar 25, 2026

Conversation

Amar3tto commented Mar 16, 2026

Uh oh!

gemini-code-assist bot commented Mar 16, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

codecov bot commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Amar3tto commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Abacn commented Mar 18, 2026

Uh oh!

Amar3tto commented Mar 18, 2026

Uh oh!

Abacn commented Mar 18, 2026

Uh oh!

Abacn commented Mar 18, 2026

Uh oh!

Abacn Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

Abacn commented Mar 19, 2026

Uh oh!

Amar3tto commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Abacn commented Mar 20, 2026

Uh oh!

Abacn commented Mar 25, 2026

Uh oh!

BentsiLeviav commented Mar 25, 2026

Uh oh!

Abacn commented Mar 25, 2026

Uh oh!

BentsiLeviav commented Mar 25, 2026

Uh oh!

BentsiLeviav commented Mar 25, 2026

Uh oh!

Abacn commented Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Mar 16, 2026 •

edited

Loading

Amar3tto commented Mar 17, 2026 •

edited

Loading

Amar3tto commented Mar 20, 2026 •

edited

Loading