Skip to content
This repository was archived by the owner on Jul 7, 2023. It is now read-only.

Commit 9d48d35

Browse files
author
cclauss
authored
Simplify native_to_unicode() & unicode_to_native()
The first [uses feature detection, instead of version detection](https://docs.python.org/3/howto/pyporting.html#use-feature-detection-instead-of-version-detection) and the second [avoids assigning a lambda expression to a variable](https://docs.quantifiedcode.com/python-anti-patterns/correctness/assigning_a_lambda_to_a_variable.html).
1 parent 2958ac1 commit 9d48d35

File tree

1 file changed

+9
-10
lines changed

1 file changed

+9
-10
lines changed

tensor2tensor/data_generators/text_encoder.py

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -56,19 +56,18 @@
5656
_ESCAPE_CHARS = set(u"\\_u;0123456789")
5757

5858

59-
def native_to_unicode_py2(s):
60-
"""Python 2: transform native string to Unicode."""
61-
return s if isinstance(s, unicode) else s.decode("utf8")
59+
def native_to_unicode(s):
60+
"""Transform native string to Unicode."""
61+
try: # Python 2
62+
return s if isinstance(s, unicode) else s.decode("utf8")
63+
except NameError: # Python 3: unicode() was dropped
64+
return s
6265

6366

6467
# Conversion between Unicode and UTF-8, if required (on Python2)
65-
if six.PY2:
66-
native_to_unicode = native_to_unicode_py2
67-
unicode_to_native = lambda s: s.encode("utf-8")
68-
else:
69-
# No conversion required on Python3
70-
native_to_unicode = lambda s: s
71-
unicode_to_native = lambda s: s
68+
def unicode_to_native(s):
69+
"""Transform Unicode to native string."""
70+
return s.encode("utf-8") if six.PY2 else s # No conversion required on Python3
7271

7372

7473
class TextEncoder(object):

0 commit comments

Comments
 (0)