v2.20.0

Release 2.20.0

Bug Fixes and Other Changes

• Fix nightly builds.
• iterate over Unicode White_Space directly, rather than testing each of 1.1M code points
• Change input size from int16_t to int32_t to support large inputs.
• Ignore SentencePiece::BYTE during encoding instead of throwing error
• Fix TF-text for TF 2.20.
• Update requirements with latest tensorflow-metadata.
• Fix configure to pull bazelrc from TF 2.20.
• update sha256 for tensorflow 2.20.0
• Revert "update sha256 for tensorflow 2.20.0"
• Change strip_prefix logic to grab sha256 properly
• Limit array-record dependency to Linux x86_64 and aarch64.
• Pin array-record to 0.4.1 for darwin platform
• Adds build_bazel_apple_support for newer macOS builds

v2.19.0

Bug Fixes and Other Changes

• Add docker build scripts and enable aarch64 pip wheels.
• Regenerate stubs with Mypy 1.13.0
• Handle the punctuation definition mismatch between different Unicode versions.
• Replace outdated select() on --cpu in core/kernels/sentencepiece/BUILD with platform API equivalent.
• cleanup of deprecated test methods
• Update tensorflow version.
• Limit dm-tree to 0.1.8.
• Remove srcs_version and python_version attributes, as they already default to "PY3"
• Remove invalid public_names_test
• Fix protobuf dependency.
• Update test files for new ICU version
• Init python_wheel_version_suffix_repository for TF dependency in tf-text project after tensorflow/tensorflow@805775f.
• Add numpy py dep and fix python dependencies.
• Add explicit tf-keras dependency.
• Update Version for 2.19.0-rc0
• Revert "Init python_wheel_version_suffix_repository for TF dependency in tf-text project after tensorflow/tensorflow@805775f."
• Refresh requirement lockfiles
• add @absl_py//absl:app dep to tensorflow_build_info
• Update Version for 2.19.0

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

C. Antonio Sánchez

v2.19.0-rc0

Bug Fixes and Other Changes

• Add docker build scripts and enable aarch64 pip wheels.
• Regenerate stubs with Mypy 1.13.0
• Handle the punctuation definition mismatch between different Unicode versions.
• Replace outdated select() on --cpu in core/kernels/sentencepiece/BUILD with platform API equivalent.
• cleanup of deprecated test methods
• Update tensorflow version.
• Limit dm-tree to 0.1.8.
• Remove srcs_version and python_version attributes, as they already default to "PY3"
• Remove invalid public_names_test
• Fix protobuf dependency.
• Update test files for new ICU version
• Init python_wheel_version_suffix_repository for TF dependency in tf-text project after tensorflow/tensorflow@805775f.
• Add numpy py dep and fix python dependencies.
• Add explicit tf-keras dependency.
• Update Version for 2.19.0-rc0
• Revert "Init python_wheel_version_suffix_repository for TF dependency in tf-text project after tensorflow/tensorflow@805775f."
• Refresh requirement lockfiles
• add @absl_py//absl:app dep to tensorflow_build_info

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

C. Antonio Sánchez

v2.18.1

Major Features and Improvements

• Add Python 3.12 and aarch64 support

Bug Fixes and Other Changes

• Add docker build scripts and enable aarch64 pip wheels.
• Replace std::string_view with absl::string_view
• Protect the fast wordpiece tokenizer from infinite looping.
• Update lock files
• Update version for 2.18.1
• Set HERMETIC_PYTHON_VERSION correctly

v2.18.0

Release 2.18.0

Bug Fixes and Other Changes

• Fix out-of-bounds read in whitespace tokenizer
• Add unit test for fixed bounds check in IsWhitespace
• Add Hermetic CUDA rules.
• Remove tf/lite/kernels/shim:tf_headers from tf/core:framework
• Update version
• Update configure.sh

v2.18.0-rc0

Release 2.18.0-rc0

Bug Fixes and Other Changes

• Fix out-of-bounds read in whitespace tokenizer
• Add unit test for fixed bounds check in IsWhitespace
• Add Hermetic CUDA rules.
• Remove tf/lite/kernels/shim:tf_headers from tf/core:framework
• Update version
• Update configure.sh

v2.17.0

Release 2.17.0

Bug Fixes and Other Changes

• negative sampling excludes positive class
• revert html encoding
• much faster set-intersection based version
• Fix notebook failure with Keras 3.
• Remove tensorflow-macos from setup.py
• Update tensorflow-macos to 2.16.1
• Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Alex Shroyer, C. Antonio Sánchez, Maggie Zhang

v2.17.0-rc0

Release 2.17.0-rc0

Bug Fixes and Other Changes

• negative sampling excludes positive class
• revert html encoding
• much faster set-intersection based version
• Fix notebook failure with Keras 3.
• Remove tensorflow-macos from setup.py
• Update tensorflow-macos to 2.16.1
• Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Alex Shroyer, C. Antonio Sánchez, Maggie Zhang

v2.16.1

Release 2.16.1

Major Features and Improvements

Breaking Changes

Bug Fixes and Other Changes

• Update tf-text setup scripts.
• Support resource manager scoped Sentencepiece resources.
• Remove use_unique_shared_resource_name.
• Remove tensorflow_text dependency on tf_hub library.
• Fix TF patch, bump TF commit
• Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Raviteja Gorijala

v2.16.0-rc0

Release 2.16.0-rc0

Major Features and Improvements

Breaking Changes

Bug Fixes and Other Changes

• Update tf-text setup scripts.
• Support resource manager scoped Sentencepiece resources.
• Remove use_unique_shared_resource_name.
• Remove tensorflow_text dependency on tf_hub library.
• Fix TF patch, bump TF commit
• Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Raviteja Gorijala