tools for model exporting and converting (#639)

* Add tools bash for model exporting and converting

* convert test

* convert test

* convert test

* produce python tool for export and convert

---------

Co-authored-by: horcham <zhanghongquan15@huawei.com>

Author: horcham

docs/cn/tutorials/frequently_asked_questions.md

@@ -9,6 +9,7 @@
  - [推理相关问题](#q8-推理时相关问题)
  - [DBNet训练速率不及预期](#q9-DBNet训练速率不及预期)
  - [`libgomp-d22c30c5.so.1.0.0`相关错误](#q10-libgomp-d22c30c5so100相关错误)
+ - [当在lmdb dataset上训练abinet报数据管道错误](#q11-当在lmdb-dataset上训练abinet报数据管道错误)
 
 ### Q1 未定义符号
 
@@ -694,3 +695,50 @@ ImportError: /root/mindocr_env/lib/python3.8/site-packages/sklearn/__check_build
    ```bash
    export LD_PRELOAD=/root/mindocr_env/lib/python3.8/site-packages/scikit_learn.libs/libgomp-d22c30c5.so.1.0.0:$LD_PRELOAD
    ```
+
+### Q11 当在lmdb dataset上训练abinet报数据管道错误
+当在lmdb dataset上训练abinet报以下数据管道错误
+```bash
+mindocr.data.rec_lmdb_dataset WARNING - Error occurred during preprocess.
+ Exception thrown from dataset pipeline. Refer to 'Dataset Pipeline Error Message'.
+
+------------------------------------------------------------------
+- Dataset Pipeline Error Message:
+------------------------------------------------------------------
+[ERROR] No cast for the specified DataType was found.
+
+------------------------------------------------------------------
+- C++ Call Stack: (For framework developers)
+------------------------------------------------------------------
+mindspore/ccsrc/minddata/dataset/kernels/py_func_op.cc(143).
+```
+可以尝试用如下步骤修复
+ - 找到mindspore的包路径
+ - 打开文件: `mindspore/dataset/transforms/transform.py`
+ - 跳转到93行，可以得到如下内容:
+  ```bash
+  93        if key in EXECUTORS_LIST:
+  94           # get the executor by process id and thread id
+  95            executor = EXECUTORS_LIST[key]
+  96            # remove the old transform which in executor and update the new transform
+  97            executor.UpdateOperation(self.parse())
+  98        else:
+  99            # create a new executor by process id and thread_id
+  100           executor = cde.Execute(self.parse())
+  101           # add the executor the global EXECUTORS_LIST
+  102           EXECUTORS_LIST[key] = executor
+  ```
+ - 使用`executor = cde.Execute(self.parse())`替换97行, 得到如下内容:
+  ```bash
+  93        if key in EXECUTORS_LIST:
+  94            # get the executor by process id and thread id
+  95            executor = EXECUTORS_LIST[key]
+  96            # remove the old transform which in executor and update the new transform
+  97            executor = cde.Execute(self.parse())
+  98        else:
+  99            # create a new executor by process id and thread_id
+  100           executor = cde.Execute(self.parse())
+  101           # add the executor the global EXECUTORS_LIST
+  102           EXECUTORS_LIST[key] = executor
+  ```
+  - 保存后再次尝试训练即可

docs/en/tutorials/frequently_asked_questions.md

@@ -8,7 +8,8 @@
  - [Problems related to model converting](#q7-problems-related-to-model-converting)
  - [Problems related to inference](#q8-problems-related-to-inference)
  - [Training speed of DBNet not as fast as expexted](#q9-training-speed-of-dbnet-not-as-fast-as-expexted)
- - [Error about `libgomp-d22c30c5.so.1.0.0`](#q9-error-about-libgomp-d22c30c5so100)
+ - [Error about `libgomp-d22c30c5.so.1.0.0`](#q10-error-about-libgomp-d22c30c5so100)
+ - [Dataset Pipeline Error when training abinet on lmdb dataset](#q11-dataset-pipeline-error-when-training-abinet-on-lmdb-dataset)
 
 ### Q1 Undefined symbol
 
@@ -683,3 +684,50 @@ You can try the following steps to fix it:
    ```bash
    export LD_PRELOAD=/root/mindocr_env/lib/python3.8/site-packages/scikit_learn.libs/libgomp-d22c30c5.so.1.0.0:$LD_PRELOAD
    ```
+
+### Q11 Dataset Pipeline Error when training abinet on lmdb dataset
+The following error may occur when training abinet on lmdb dataset
+```bash
+mindocr.data.rec_lmdb_dataset WARNING - Error occurred during preprocess.
+ Exception thrown from dataset pipeline. Refer to 'Dataset Pipeline Error Message'.
+
+------------------------------------------------------------------
+- Dataset Pipeline Error Message:
+------------------------------------------------------------------
+[ERROR] No cast for the specified DataType was found.
+
+------------------------------------------------------------------
+- C++ Call Stack: (For framework developers)
+------------------------------------------------------------------
+mindspore/ccsrc/minddata/dataset/kernels/py_func_op.cc(143).
+```
+You can try the following steps to fix it:
+ - find the folder of mindspore package
+ - open file: `mindspore/dataset/transforms/transform.py`
+ - switch to line 93:
+  ```bash
+  93        if key in EXECUTORS_LIST:
+  94           # get the executor by process id and thread id
+  95            executor = EXECUTORS_LIST[key]
+  96            # remove the old transform which in executor and update the new transform
+  97            executor.UpdateOperation(self.parse())
+  98        else:
+  99            # create a new executor by process id and thread_id
+  100           executor = cde.Execute(self.parse())
+  101           # add the executor the global EXECUTORS_LIST
+  102           EXECUTORS_LIST[key] = executor
+  ```
+ - replace line 97 with `executor = cde.Execute(self.parse())`, and get
+  ```bash
+  93        if key in EXECUTORS_LIST:
+  94            # get the executor by process id and thread id
+  95            executor = EXECUTORS_LIST[key]
+  96            # remove the old transform which in executor and update the new transform
+  97            executor = cde.Execute(self.parse())
+  98        else:
+  99            # create a new executor by process id and thread_id
+  100           executor = cde.Execute(self.parse())
+  101           # add the executor the global EXECUTORS_LIST
+  102           EXECUTORS_LIST[key] = executor
+  ```
+  - save the file, and try to train the model.

mindocr/models/det_dbnet.py

@@ -19,8 +19,12 @@ def _cfg(url='', **kwargs):
     'dbnet_resnet18': _cfg(
         url='https://download.mindspore.cn/toolkits/mindocr/dbnet/dbnet_resnet18-0c0c4cfa.ckpt'),
     'dbnet_resnet50': _cfg(
+        url='https://download.mindspore.cn/toolkits/mindocr/dbnet/dbnet_resnet50-c3a4aa24.ckpt'),
+    'dbnet_resnet50_ch_en': _cfg(
         url='https://download.mindspore.cn/toolkits/mindocr/dbnet/dbnet_resnet50_ch_en_general-a5dbb141.ckpt'),
     'dbnetpp_resnet50': _cfg(
+        url='https://download-mindspore.osinfra.cn/toolkits/mindocr/dbnet/dbnetpp_resnet50-068166c2.ckpt'),
+    'dbnetpp_resnet50_ch_en': _cfg(
         url='https://download.mindspore.cn/toolkits/mindocr/dbnet/dbnetpp_resnet50_ch_en_general-884ba5b9.ckpt'),
     'dbnet_ppocrv3': _cfg(
         url='https://download-mindspore.osinfra.cn/toolkits/mindocr/dbnet/dbnet_mobilenetv3_ppocrv3-70d6018f.ckpt')
@@ -125,6 +129,35 @@ def dbnet_resnet50(pretrained=False, pretrained_backbone=True, **kwargs):
     return model
 
 
+@register_model
+def dbnet_resnet50_ch_en(pretrained=False, pretrained_backbone=True, **kwargs):
+    model_config = {
+        "backbone": {
+            'name': 'det_resnet50',
+            'pretrained': pretrained_backbone
+        },
+        "neck": {
+            "name": 'DBFPN',
+            "out_channels": 256,
+            "bias": False,
+        },
+        "head": {
+            "name": 'DBHead',
+            "k": 50,
+            "bias": False,
+            "adaptive": True
+        }
+    }
+    model = DBNet(model_config)
+
+    # load pretrained weights
+    if pretrained:
+        default_cfg = default_cfgs['dbnet_resnet50_ch_en']
+        load_pretrained(model, default_cfg)
+
+    return model
+
+
 @register_model
 def dbnetpp_resnet50(pretrained=False, pretrained_backbone=True, **kwargs):
     model_config = {
@@ -156,6 +189,37 @@ def dbnetpp_resnet50(pretrained=False, pretrained_backbone=True, **kwargs):
     return model
 
 
+@register_model
+def dbnetpp_resnet50_ch_en(pretrained=False, pretrained_backbone=True, **kwargs):
+    model_config = {
+        "backbone": {
+            'name': 'det_resnet50',
+            'pretrained': pretrained_backbone
+        },
+        "neck": {
+            "name": 'DBFPN',
+            "out_channels": 256,
+            "bias": False,
+            "use_asf": True,
+            "channel_attention": True
+        },
+        "head": {
+            "name": 'DBHead',
+            "k": 50,
+            "bias": False,
+            "adaptive": True
+        }
+    }
+    model = DBNet(model_config)
+
+    # load pretrained weights
+    if pretrained:
+        default_cfg = default_cfgs['dbnetpp_resnet50_ch_en']
+        load_pretrained(model, default_cfg)
+
+    return model
+
+
 @register_model
 def dbnet_ppocrv3(pretrained=False, **kwargs):
     model_config = {

mindocr/models/rec_abinet.py

@@ -12,7 +12,7 @@ def _cfg(url="", **kwargs):
 default_cfgs = {
     # 'abinet':
     "abinet": _cfg(
-        url="https://download.mindspore.cn/toolkits/mindocr/abinet/abinet_pretrain_en-821ca20b.ckpt"
+        url="https://download-mindspore.osinfra.cn/toolkits/mindocr/abinet/abinet_resnet45_en-7efa1184.ckpt"
     ),
 }