Skip to content

Upgrade protobuf to match with tf-nightly #170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mtsokol wants to merge 1 commit into from
Closed

Upgrade protobuf to match with tf-nightly #170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions MODULE.bazel
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,11 @@ module(
)

bazel_dep(name = "rules_proto", version = "7.0.2")
bazel_dep(name = "rules_python", version = "0.37.0")
bazel_dep(name = "platforms", version = "0.0.10")
bazel_dep(name = "protobuf", version = "28.3")
bazel_dep(name = "rules_python", version = "1.0.0")
bazel_dep(name = "platforms", version = "0.0.11")
bazel_dep(name = "protobuf", version = "31.1")
bazel_dep(name = "googletest", version = "1.15.2")
bazel_dep(name = "abseil-cpp", version = "20240722.0")
bazel_dep(name = "abseil-cpp", version = "20250127.0")
bazel_dep(name = "abseil-py", version = "2.1.0")
bazel_dep(name = "eigen", version = "3.4.0.bcr.3")
bazel_dep(name = "riegeli", version = "0.0.0-20241218-3385e3c")
Expand Down
2 changes: 0 additions & 2 deletions beam/examples/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1 @@
array-record[beam]
google-cloud-storage==2.11.0
tensorflow==2.14.0
4 changes: 2 additions & 2 deletions cpp/array_record_writer.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,11 +217,11 @@ class ArrayRecordWriterBase : public riegeli::Object {
// 31 in 64-bit build).
//
// Default: `absl::nullopt`.
Options& set_window_log(std::optional<int> window_log) {
Options& set_window_log(absl::optional<int> window_log) {
compressor_options_.set_window_log(window_log);
Copy link
Contributor

@dryman dryman Aug 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The upstream riegeli package uses std::optional instead of absl::optional.
https://github.com/google/riegeli/blob/0895cfaf4787a03afeed86ddacba3dc9a8beae52/riegeli/brotli/brotli_writer.h#L87

So I prefer to use the same convention.

Copy link
Contributor Author

@mtsokol mtsokol Aug 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure! I did it because for some reason on my local macos arm setup this was causing a compilation error.

Copy link
Contributor

@dryman dryman Aug 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may due to two reasons (either or, or both)

  1. The compiler didn't recognize c++17 (I think this is less likely because we already specify --cxxopt=std=c++17)
  2. Linking incorrect stdlib (in some platforms, there are separated libstdc++)

You might find you installed libstdc++ on your platform which didn't support c++17, but the binary linked to it.

return *this;
}
std::optional<int> window_log() const {
absl::optional<int> window_log() const {
return compressor_options_.window_log();
}

Expand Down
4 changes: 2 additions & 2 deletions oss/build_whl.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,12 +87,12 @@ function main() {
echo $(date) : "=== Output wheel file is in: ${DEST}"

# Install ArrayRecord from the wheel and run smoke tests.
$PYTHON_BIN -m pip install --find-links="${DEST}" --pre array-record
$PYTHON_BIN -m pip install ${OUTPUT_DIR}/all_dist/array_record*.whl
$PYTHON_BIN -c 'import array_record'
$PYTHON_BIN -c 'from array_record.python import array_record_data_source'
# TF is not available on Python 3.13 and above.
if [ "$(uname)" != "Darwin" ] && (( "${PYTHON_MINOR_VERSION}" < 13 )); then
$PYTHON_BIN -m pip install jax tensorflow grain
$PYTHON_BIN -m pip install jax tensorflow==2.20.0rc0 grain
$PYTHON_BIN oss/test_import_grain.py
$PYTHON_BIN oss/test_import_tensorflow.py
fi
Expand Down
16 changes: 12 additions & 4 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,19 @@
'etils[epath]',
]

TF_PACKAGE = [
'tensorflow>=2.20.0rc0'
]

BEAM_EXTRAS = [
'apache-beam[gcp]==2.53.0',
'apache-beam[gcp]>=2.53.0',
'google-cloud-storage>=2.11.0',
'tensorflow>=2.14.0'
]
] + TF_PACKAGE

TEST_EXTRAS = [
'jax',
'grain',
] + TF_PACKAGE


class BinaryDistribution(Distribution):
Expand All @@ -34,7 +42,7 @@ def has_ext_modules(self):
package_data={'': ['*.so']},
python_requires='>=3.10',
install_requires=REQUIRED_PACKAGES,
extras_require={'beam': BEAM_EXTRAS},
extras_require={'beam': BEAM_EXTRAS, 'test': TEST_EXTRAS},
url='https://github.com/google/array_record',
license='Apache-2.0',
classifiers=[
Expand Down