Skip to content

Commit 528d695

Browse files
authored
[CMake] Enable SentencePiece tokenizer by default (#47)
The SentencePiece tokenizer was disabled by default to reduce the binary size of the built library, while it causes some error when users expect to use the SentencePiece tokenizer. This PR enables it by default. And we will need to manually disable it if we need to reduce its binary size.
1 parent 5de6f65 commit 528d695

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

CMakeLists.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ set(TOKENIZERS_CPP_CARGO_SOURCE_PATH ${TOKENIZERS_CPP_ROOT}/rust)
8888
option(MSGPACK_USE_BOOST "Use Boost libraried" OFF)
8989
add_subdirectory(msgpack)
9090

91-
option(MLC_ENABLE_SENTENCEPIECE_TOKENIZER "Enable SentencePiece tokenizer" OFF)
91+
option(MLC_ENABLE_SENTENCEPIECE_TOKENIZER "Enable SentencePiece tokenizer" ON)
9292

9393
if(MSVC)
9494
set(TOKENIZERS_RUST_LIB "${TOKENIZERS_CPP_CARGO_BINARY_DIR}/tokenizers_c.lib")

0 commit comments

Comments
 (0)