Skip to content

Commit 11d0d0c

Browse files
authored
Remove old comment. (#230)
specialTokens are now conditionally ignored on skipSpecialTokens at decode time.
1 parent f000aa7 commit 11d0d0c

File tree

1 file changed

+0
-1
lines changed

1 file changed

+0
-1
lines changed

Sources/Tokenizers/Tokenizer.swift

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -321,7 +321,6 @@ public class PreTrainedTokenizer: Tokenizer {
321321
}.joined(separator: "|")
322322
addedTokensRegex = try? NSRegularExpression(pattern: addedTokensRegexString, options: [])
323323

324-
// TODO: specialTokens are stored but never used
325324
self.specialTokens = specialTokens
326325
self.addedTokens = Set(addedTokens.keys)
327326

0 commit comments

Comments
 (0)