Skip to content

Commit 160785c

Browse files
authored
Fix tokenization of codes containing UTF-8 characters (#122)
1 parent fe51fe6 commit 160785c

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

src/TextMate/Tokenizer.php

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ public function tokenize(string $text): array
4545
contentNameScopesList: $scopeList,
4646
);
4747

48-
$lines = array_values(preg_split("/\R/", $text));
48+
$lines = array_values(preg_split("/\R/u", $text));
4949
$tokens = [];
5050

5151
foreach ($lines as $index => $line) {

0 commit comments

Comments
 (0)