Skip to content

Commit bc6e6cd

Browse files
authored
fix: used normalize_str in BaseTokenizer.normalize (#1884)
1 parent d6a4acc commit bc6e6cd

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

bindings/python/py_src/tokenizers/implementations/base_tokenizer.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -187,7 +187,7 @@ def normalize(self, sequence: str) -> str:
187187
Returns:
188188
The normalized string
189189
"""
190-
return self._tokenizer.normalize(sequence)
190+
return self._tokenizer.normalizer.normalize_str(sequence)
191191

192192
def encode(
193193
self,

0 commit comments

Comments
 (0)