Skip to content

Commit df37192

Browse files
fatcat-zfatcat-z
authored
Update the dtype since latest version of Numpy has removed str. (#2112)
* Update the dtype since latest version of Numpy has removed str. * Set Numpy to a fixed version in CI. * Update the protobuf's version in CI. * Keep h5py to a lower version for tf 1.15 in CI. Signed-off-by: Jay Zhang <jiz@microsoft.com>
1 parent 434b4a7 commit df37192

12 files changed

+82
-71
lines changed

‎ci_build/azure_pipelines/keras2onnx_application_tests.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ jobs:
1818
INSTALL_KERAS_RESNET: pip install keras-resnet
1919
INSTALL_TRANSFORMERS:
2020
INSTALL_NUMPY: pip install numpy==1.19.0
21+
INSTALL_LEGACY: pip install h5py==2.9.0
2122
NIGHTLY_BUILD_TEST: python run_all_v2.py --exclude "test_keras_applications_v2.py"
2223

2324
Python37-onnx1.11-tf2.5:
@@ -77,6 +78,7 @@ jobs:
7778
INSTALL_KERAS_RESNET: pip install keras-resnet
7879
INSTALL_TRANSFORMERS:
7980
INSTALL_NUMPY: pip install numpy==1.19.0
81+
INSTALL_LEGACY: pip install h5py==2.9.0
8082
NIGHTLY_BUILD_TEST: python run_all_v2.py --exclude "test_keras_applications_v2.py"
8183

8284
Python37-onnx1.11-tf2.5:

‎ci_build/azure_pipelines/templates/keras2onnx_application_tests.yml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ steps:
2121
pip install $(ONNX_PATH)
2222
pip uninstall -y protobuf
2323
pip install "protobuf<4.21.0"
24-
pip install h5py==2.9.0
24+
pip install h5py==3.7.0
2525
pip install parameterized
2626
pip install coloredlogs flatbuffers
2727
$(INSTALL_TENSORFLOW)
@@ -48,6 +48,10 @@ steps:
4848
pip install keras-self-attention
4949
pip install pytest pytest-cov pytest-runner
5050
$(INSTALL_NUMPY)
51+
if [[ ! -z $INSTALL_LEGACY ]];
52+
then
53+
pip install $(INSTALL_LEGACY)
54+
fi
5155
5256
pip freeze --all
5357
displayName: 'Install dependencies'
@@ -85,7 +89,7 @@ steps:
8589
pip install %ONNX_PATH%
8690
pip uninstall -y protobuf
8791
pip install "protobuf<4.21.0"
88-
pip install h5py==2.9.0
92+
pip install h5py==3.7.0
8993
pip install parameterized
9094
pip install coloredlogs flatbuffers
9195
%INSTALL_TENSORFLOW%
@@ -107,6 +111,7 @@ steps:
107111
pip install keras-self-attention
108112
pip install pytest pytest-cov pytest-runner
109113
%INSTALL_NUMPY%
114+
IF NOT "%INSTALL_LEGACY%"=="" (pip install %INSTALL_LEGACY%)
110115
111116
pip freeze --all
112117
displayName: 'Install dependencies'

‎ci_build/azure_pipelines/templates/keras2onnx_unit_test.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ steps:
2121
pip install $(ONNX_PATH)
2222
pip uninstall -y protobuf
2323
pip install "protobuf<4.21.0"
24-
pip install h5py==2.9.0
24+
pip install h5py==3.7.0
2525
pip install parameterized
2626
pip install coloredlogs flatbuffers
2727
pip install $(TENSORFLOW_PATH)
@@ -75,7 +75,7 @@ steps:
7575
pip install %ONNX_PATH%
7676
pip uninstall -y protobuf
7777
pip install "protobuf<4.21.0"
78-
pip install h5py==2.9.0
78+
pip install h5py==3.7.0
7979
pip install parameterized
8080
pip install coloredlogs flatbuffers
8181
pip install %TENSORFLOW_PATH%

‎ci_build/azure_pipelines/templates/setup.yml‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,18 +6,12 @@ steps:
66
pip install pytest pytest-cov pytest-runner coverage graphviz requests pyyaml pillow pandas parameterized sympy coloredlogs flatbuffers
77
pip install $(CI_PIP_TF_NAME) $(CI_PIP_ONNX_NAME)
88
9-
# Protobuf 3.20 results in linker errors on Windows in TF.
10-
# Protobuf 4.0 is binary incompatible with what C++ TF uses.
11-
# https://github.com/tensorflow/tensorflow/blob/c3337c73306b2b859d82fe130912f18e6a1c5c23/tensorflow/tools/pip_package/setup.py#L88
12-
pip uninstall -y protobuf
13-
pip install "protobuf<3.20.0"
14-
159
# TF < 2.7 reuires numpy <= 1.19, but onnxruntime >= 1.11 requires numpy >= 1.21
1610
if [[ $CI_TF_VERSION < 2.7 ]] && [[ $CI_ONNX_BACKEND == "onnxruntime" ]] ;
1711
then
1812
pip install $(CI_PIP_ONNX_BACKEND_NAME) numpy --no-deps -U
1913
else
20-
pip install $(CI_PIP_ONNX_BACKEND_NAME)
14+
pip install $(CI_PIP_ONNX_BACKEND_NAME) numpy==1.23.4
2115
fi
2216
# TF 1.10 requires numpy <=1.14.5 and >=1.13.3, but onnxruntime 0.2.1 does not work with numpy <= 1.14.5
2317
# Upgrade numpy only within constraints from other packages if any.
@@ -80,6 +74,12 @@ steps:
8074
fi
8175
fi
8276
77+
# Protobuf 3.20 results in linker errors on Windows in TF.
78+
# Protobuf 4.0 is binary incompatible with what C++ TF uses.
79+
# https://github.com/tensorflow/tensorflow/blob/c3337c73306b2b859d82fe130912f18e6a1c5c23/tensorflow/tools/pip_package/setup.py#L88
80+
pip uninstall -y protobuf
81+
pip install "protobuf==3.20.3"
82+
8383
python setup.py install
8484
8585
if [[ $ONNX_TEST == "true" ]] ;

‎ci_build/azure_pipelines/trimmed_keras2onnx_application_tests.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ jobs:
1818
INSTALL_KERAS_RESNET: pip install keras-resnet
1919
INSTALL_TRANSFORMERS:
2020
INSTALL_NUMPY: pip install numpy==1.19.0
21+
INSTALL_LEGACY: pip install h5py==2.9.0
2122
NIGHTLY_BUILD_TEST: python run_all.py --exclude "test_keras_applications_v2.py"
2223

2324
Python38-tf2-latest:

‎ci_build/azure_pipelines/unit_test.yml‎

Lines changed: 36 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,18 @@
33
stages:
44
- stage:
55
jobs:
6-
- template: 'templates/job_generator.yml'
7-
parameters:
8-
python_versions: ['3.9']
9-
tf_versions: ['2.8.0']
10-
onnx_opsets: ['']
11-
skip_tfjs_tests: 'False'
12-
skip_tf_tests: 'True'
13-
job:
14-
steps:
15-
- template: 'unit_test.yml'
16-
report_coverage: 'True'
6+
# TODO: Enable these tests once https://github.com/onnx/tensorflow-onnx/issues/2118 is fixed.
7+
# - template: 'templates/job_generator.yml'
8+
# parameters:
9+
# python_versions: ['3.9']
10+
# tf_versions: ['2.8.0']
11+
# onnx_opsets: ['']
12+
# skip_tfjs_tests: 'False'
13+
# skip_tf_tests: 'True'
14+
# job:
15+
# steps:
16+
# - template: 'unit_test.yml'
17+
# report_coverage: 'True'
1718

1819
- template: 'templates/job_generator.yml'
1920
parameters:
@@ -37,17 +38,18 @@ stages:
3738
- template: 'unit_test.yml'
3839
report_coverage: 'True'
3940

40-
- template: 'templates/job_generator.yml'
41-
parameters:
42-
python_versions: ['3.9']
43-
tf_versions: ['2.7.4']
44-
onnx_opsets: ['']
45-
skip_tfjs_tests: 'False'
46-
skip_tf_tests: 'True'
47-
job:
48-
steps:
49-
- template: 'unit_test.yml'
50-
report_coverage: 'True'
41+
# TODO: Enable these tests once https://github.com/onnx/tensorflow-onnx/issues/2118 is fixed.
42+
# - template: 'templates/job_generator.yml'
43+
# parameters:
44+
# python_versions: ['3.9']
45+
# tf_versions: ['2.7.4']
46+
# onnx_opsets: ['']
47+
# skip_tfjs_tests: 'False'
48+
# skip_tf_tests: 'True'
49+
# job:
50+
# steps:
51+
# - template: 'unit_test.yml'
52+
# report_coverage: 'True'
5153

5254
- template: 'templates/job_generator.yml'
5355
parameters:
@@ -61,17 +63,18 @@ stages:
6163
- template: 'unit_test.yml'
6264
report_coverage: 'True'
6365

64-
- template: 'templates/job_generator.yml'
65-
parameters:
66-
python_versions: ['3.10']
67-
tf_versions: ['2.9.1']
68-
onnx_opsets: ['']
69-
skip_tfjs_tests: 'False'
70-
skip_tf_tests: 'True'
71-
job:
72-
steps:
73-
- template: 'unit_test.yml'
74-
report_coverage: 'True'
66+
# TODO: Enable these tests once https://github.com/onnx/tensorflow-onnx/issues/2118 is fixed.
67+
# - template: 'templates/job_generator.yml'
68+
# parameters:
69+
# python_versions: ['3.10']
70+
# tf_versions: ['2.9.1']
71+
# onnx_opsets: ['']
72+
# skip_tfjs_tests: 'False'
73+
# skip_tf_tests: 'True'
74+
# job:
75+
# steps:
76+
# - template: 'unit_test.yml'
77+
# report_coverage: 'True'
7578

7679
- template: 'templates/job_generator.yml'
7780
parameters:

‎tests/run_pretrained_models.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -525,7 +525,7 @@ def run_tflite():
525525
inputs[k] = np_value.astype(expected_dtype)
526526
else:
527527
if expected_dtype == "string":
528-
inputs[k] = self.make_input(v).astype(np.str).astype(object)
528+
inputs[k] = self.make_input(v).astype(str).astype(object)
529529
else:
530530
inputs[k] = self.make_input(v).astype(expected_dtype)
531531

‎tests/test_api.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -103,7 +103,7 @@ def test_keras_hashtable(self):
103103
model.compile(optimizer='adam', loss=tf.keras.losses.mean_squared_error)
104104

105105
inp1 = np.array([[2.], [3.]], dtype=np.float32)
106-
inp2 = np.array([["a"], ["b"]], dtype=np.str)
106+
inp2 = np.array([["a"], ["b"]], dtype=str)
107107
if not is_tf2():
108108
tf.keras.backend.get_session().run(tf.tables_initializer(name='init_all_tables'))
109109
k_res = model.predict([inp1, inp2])

‎tests/test_backend.py‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1648,8 +1648,8 @@ def func(x1, x2, x3):
16481648
self._run_test_case(func, [_OUTPUT], {_INPUT: x_val1, _INPUT1: x_val2, "input3:0": x_val3})
16491649

16501650
def test_concat_const_string(self):
1651-
x_val1 = np.array([["Hello world", "abc"], ["def", "♦♥♠♣"]], dtype=np.str)
1652-
const_val = np.array([["Hello there", "wxyz"], ["", "π"]], dtype=np.str)
1651+
x_val1 = np.array([["Hello world", "abc"], ["def", "♦♥♠♣"]], dtype=str)
1652+
const_val = np.array([["Hello there", "wxyz"], ["", "π"]], dtype=str)
16531653
def func(x1):
16541654
x_ = tf.concat([x1, const_val], 0)
16551655
return tf.identity(x_, name=_TFOUTPUT)
@@ -3513,9 +3513,9 @@ def func(x1, x2):
35133513
def test_where_string(self):
35143514
x_val = np.array([1, 2, -3, 4, -5, -6, -7, 8, 9, 0], dtype=np.float32)
35153515
true_result = np.array([111, 222, 333, 444, 555, 666, 777, 888, 999, 1000],
3516-
dtype=np.str)
3516+
dtype=str)
35173517
false_result = np.array([-111, -222, -333, -444, -555, -666, -777, -888, -999, -1000],
3518-
dtype=np.str)
3518+
dtype=str)
35193519
def func(x):
35203520
picks = tf.where(x > -1, true_result, false_result)
35213521
return tf.identity(picks, name=_TFOUTPUT)
@@ -3613,7 +3613,7 @@ def func(x):
36133613
@check_tf_min_version("1.14", "tf.strings.lower")
36143614
@check_opset_min_version(10, "StringNormalizer")
36153615
def test_string_lower(self):
3616-
text_val1 = np.array([["a", "Test 1 2 3", "♠♣"], ["Hi there", "test test", "♥♦"]], dtype=np.str)
3616+
text_val1 = np.array([["a", "Test 1 2 3", "♠♣"], ["Hi there", "test test", "♥♦"]], dtype=str)
36173617
def func(text1):
36183618
x = tf.strings.lower(text1)
36193619
x_ = tf.identity(x, name=_TFOUTPUT)
@@ -3623,7 +3623,7 @@ def func(text1):
36233623
@check_tf_min_version("1.14", "tf.strings.lower")
36243624
@check_opset_min_version(10, "StringNormalizer")
36253625
def test_string_lower_flat(self):
3626-
text_val1 = np.array(["a", "Test 1 2 3", "♠♣", "Hi there", "test test", "♥♦"], dtype=np.str)
3626+
text_val1 = np.array(["a", "Test 1 2 3", "♠♣", "Hi there", "test test", "♥♦"], dtype=str)
36273627
def func(text1):
36283628
x = tf.strings.lower(text1)
36293629
x_ = tf.identity(x, name=_TFOUTPUT)
@@ -3633,7 +3633,7 @@ def func(text1):
36333633
@check_tf_min_version("1.14", "tf.strings.lower")
36343634
@check_opset_min_version(10, "StringNormalizer")
36353635
def test_string_upper(self):
3636-
text_val1 = np.array([["a", "Test 1 2 3", "♠♣"], ["Hi there", "test test", "♥♦"]], dtype=np.str)
3636+
text_val1 = np.array([["a", "Test 1 2 3", "♠♣"], ["Hi there", "test test", "♥♦"]], dtype=str)
36373637
def func(text1):
36383638
x = tf.strings.upper(text1)
36393639
x_ = tf.identity(x, name=_TFOUTPUT)
@@ -4709,9 +4709,9 @@ def func(x, y, z):
47094709
@check_opset_min_version(11, "ScatterND")
47104710
@skip_tflite("Conversion crashes")
47114711
def test_tensor_scatter_update_str(self):
4712-
x_val = np.array(['A', '♠♣♥♦', 'B', 'C'], dtype=np.str).reshape((4))
4712+
x_val = np.array(['A', '♠♣♥♦', 'B', 'C'], dtype=str).reshape((4))
47134713
y_val = np.array([0, 2], dtype=np.int64).reshape((2, 1))
4714-
z_val = np.array(['☺', '11'], dtype=np.str).reshape((2))
4714+
z_val = np.array(['☺', '11'], dtype=str).reshape((2))
47154715

47164716
def func(x, y, z):
47174717
x_ = tf.tensor_scatter_nd_update(x, y, z)
@@ -4722,9 +4722,9 @@ def func(x, y, z):
47224722
@check_opset_min_version(11, "ScatterND")
47234723
@skip_tflite("Conversion crashes")
47244724
def test_tensor_scatter_update_str_const(self):
4725-
x_val = np.array(['A', '♠♣♥♦', 'B', 'C'], dtype=np.str).reshape((4))
4725+
x_val = np.array(['A', '♠♣♥♦', 'B', 'C'], dtype=str).reshape((4))
47264726
y_val = np.array([0, 2], dtype=np.int64).reshape((2, 1))
4727-
z_val = np.array(['☺', '11'], dtype=np.str).reshape((2))
4727+
z_val = np.array(['☺', '11'], dtype=str).reshape((2))
47284728

47294729
def func(x, y):
47304730
z = tf.constant(z_val)

‎tests/test_string_ops.py‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ class StringOpsTests(Tf2OnnxBackendTestBase):
3535

3636
@requires_custom_ops("StringRegexReplace")
3737
def test_static_regex_replace(self):
38-
text_val = np.array([["Hello world!", "Test 1 2 3"], ["Hi there", "test test"]], dtype=np.str)
38+
text_val = np.array([["Hello world!", "Test 1 2 3"], ["Hi there", "test test"]], dtype=str)
3939
def func(text):
4040
x_ = tf.strings.regex_replace(text, " ", "_", replace_global=True)
4141
return tf.identity(x_, name=_TFOUTPUT)
@@ -44,9 +44,9 @@ def func(text):
4444
@requires_custom_ops("StringJoin")
4545
@check_opset_min_version(8, "Expand")
4646
def test_string_join(self):
47-
text_val1 = np.array([["a", "Test 1 2 3"], ["Hi there", "test test"]], dtype=np.str)
48-
text_val2 = np.array([["b", "Test 1 2 3"], ["Hi there", "suits ♠♣♥♦"]], dtype=np.str)
49-
text_val3 = np.array("Some scalar text", dtype=np.str)
47+
text_val1 = np.array([["a", "Test 1 2 3"], ["Hi there", "test test"]], dtype=str)
48+
text_val2 = np.array([["b", "Test 1 2 3"], ["Hi there", "suits ♠♣♥♦"]], dtype=str)
49+
text_val3 = np.array("Some scalar text", dtype=str)
5050
def func(text1, text2, text3):
5151
x_ = tf.strings.join([text1, text2, text3], separator="±")
5252
return tf.identity(x_, name=_TFOUTPUT)
@@ -55,7 +55,7 @@ def func(text1, text2, text3):
5555
@requires_custom_ops("StringSplit")
5656
@check_tf_min_version("2.0", "result is sparse not ragged in tf1")
5757
def test_string_split(self):
58-
text_val = np.array([["a", "Test 1 2 3"], ["Hi there", "test test"]], dtype=np.str)
58+
text_val = np.array([["a", "Test 1 2 3"], ["Hi there", "test test"]], dtype=str)
5959
def func(text):
6060
x = tf.strings.split(text, sep=' ').flat_values
6161
x_ = tf.identity(x, name=_TFOUTPUT)
@@ -64,7 +64,7 @@ def func(text):
6464

6565
@requires_custom_ops("StringToHashBucketFast")
6666
def test_string_to_hash_bucket_fast(self):
67-
text_val = np.array([["a", "Test 1 2 3", "♠♣"], ["Hi there", "test test", "♥♦"]], dtype=np.str)
67+
text_val = np.array([["a", "Test 1 2 3", "♠♣"], ["Hi there", "test test", "♥♦"]], dtype=str)
6868
def func(text):
6969
x = tf.strings.to_hash_bucket_fast(text, 20)
7070
x_ = tf.identity(x, name=_TFOUTPUT)
@@ -73,8 +73,8 @@ def func(text):
7373

7474
@requires_custom_ops("StringEqual")
7575
def test_string_equal(self):
76-
text_val1 = np.array([["a", "Test 1 2 3", "♠♣"], ["Hi there", "test test", "♥♦"]], dtype=np.str)
77-
text_val2 = np.array([["a", "Test 2 4 6", "♠♣"], ["Hello", "test test", "♥ ♦"]], dtype=np.str)
76+
text_val1 = np.array([["a", "Test 1 2 3", "♠♣"], ["Hi there", "test test", "♥♦"]], dtype=str)
77+
text_val2 = np.array([["a", "Test 2 4 6", "♠♣"], ["Hello", "test test", "♥ ♦"]], dtype=str)
7878
def func(text1, text2):
7979
x = tf.equal(text1, text2)
8080
x_ = tf.identity(x, name=_TFOUTPUT)
@@ -83,8 +83,8 @@ def func(text1, text2):
8383

8484
@requires_custom_ops("StringNotEqual")
8585
def test_string_not_equal(self):
86-
text_val1 = np.array([["a", "Test 1 2 3", "♠♣"], ["Hi there", "test test", "♥♦"]], dtype=np.str)
87-
text_val2 = np.array([["a", "Test 2 4 6", "♠♣"], ["Hello", "test test", "♥ ♦"]], dtype=np.str)
86+
text_val1 = np.array([["a", "Test 1 2 3", "♠♣"], ["Hi there", "test test", "♥♦"]], dtype=str)
87+
text_val2 = np.array([["a", "Test 2 4 6", "♠♣"], ["Hello", "test test", "♥ ♦"]], dtype=str)
8888
def func(text1, text2):
8989
x = tf.not_equal(text1, text2)
9090
x_ = tf.identity(x, name=_TFOUTPUT)
@@ -116,7 +116,7 @@ def test_regex_split_with_offsets(self):
116116
from tensorflow_text.python.ops.regex_split_ops import (
117117
gen_regex_split_ops as lib_gen_regex_split_ops)
118118
text_val = np.array(["a Test 1 2 3 ♠♣",
119-
"Hi there test test ♥♦"], dtype=np.str)
119+
"Hi there test test ♥♦"], dtype=str)
120120
def func(text):
121121
tokens, begin_offsets, end_offsets, row_splits = lib_gen_regex_split_ops.regex_split_with_offsets(
122122
text, "(\\s)", "")
@@ -153,7 +153,7 @@ def _CreateTable(vocab, num_oov=1):
153153
init, num_oov, lookup_key_dtype=tf.string)
154154

155155
vocab = _CreateTable(["great", "they", "the", "##'", "##re", "##est"])
156-
text_val = np.array(["they're", "the", "greatest"], dtype=np.str)
156+
text_val = np.array(["they're", "the", "greatest"], dtype=str)
157157

158158
def func(text):
159159
inputs = ragged_tensor.convert_to_tensor_or_ragged_tensor(text)

0 commit comments

Comments
 (0)