Change extension on Character providing isExtendedPunctuation to private access level

mattt · mattt · commit d706d44e5fbd · 2025-09-23T09:04:15.000-07:00
diff --git a/Sources/Tokenizers/BertTokenizer.swift b/Sources/Tokenizers/BertTokenizer.swift
@@ -223,7 +223,7 @@ class BasicTokenizer {
     }
 }
 
-extension Character {
+private extension Character {
     /// https://github.com/huggingface/transformers/blob/8c1b5d37827a6691fef4b2d926f2d04fb6f5a9e3/src/transformers/tokenization_utils.py#L367
     var isExtendedPunctuation: Bool {
         if isPunctuation { return true }

Original file line number	Diff line number	Diff line change
`@@ -223,7 +223,7 @@ class BasicTokenizer {`
`223`	`223`	`}`
`224`	`224`	`}`
`225`	`225`
`226`		`-extension Character {`
	`226`	`+private extension Character {`
`227`	`227`	`/// https://github.com/huggingface/transformers/blob/8c1b5d37827a6691fef4b2d926f2d04fb6f5a9e3/src/transformers/tokenization_utils.py#L367`
`228`	`228`	`var isExtendedPunctuation: Bool {`
`229`	`229`	`if isPunctuation { return true }`