Skip to content
This repository was archived by the owner on Jul 7, 2023. It is now read-only.

Commit 63135e0

Browse files
nshazeerRyan Sepassi
authored andcommitted
Fix behavior of max chars in ChoppedTextProblem
PiperOrigin-RevId: 181708791
1 parent 20d7919 commit 63135e0

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

tensor2tensor/data_generators/problem.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1144,10 +1144,10 @@ def file_generator(self,
11441144
chars_this_file += len(text)
11451145
if text:
11461146
yield text
1147+
if max_chars_total and chars_total >= max_chars_total:
1148+
return
11471149
if max_chars_per_file and chars_this_file >= max_chars_per_file:
11481150
break
1149-
if max_chars_total and chars_total >= max_chars_total:
1150-
break
11511151

11521152
def example_generator(self, encoder, tmp_dir, task_id):
11531153
"""Generator for examples.

0 commit comments

Comments
 (0)