Legacy engine migration

Legacy engine to runtime migration

Legacy engine to runtime migration helps to migrate the ML workloads from legacy engines (to be deprecated) to ml runtimes. Below mentioned are the details for smooth migration

1. Pre-Requisite

   Create <home-directory>/.cmlutils/import-config.ini file to populate the engine to runtime mapping. Please refer this

2. Steps to populate legacy engine to runtime mapping

• Run the command

  • cmlutil helpers populate_engine_runtimes_mapping to populate the mapping
  • Above command creates <home-directory>/.cmlutils/legacy_engine_runtime_constants.json. Please make sure the tool is having necessary write permissions to create/update the file in the <home-directory>/.cmlutils/ folder.

• Export

  • Export command automatically picks up the available <home-directory>/.cmlutils/legacy_engine_runtime_constants.json and prepares the metadata accordingly.

• Import

  • Now running the import command will make sure ML Workloads with legacy engines are automatically migrated to runtimes.

3. A Sample <home-directory>/.cmlutils/legacy_engine_runtime_constants.json looks as below

      {
       "python3": "docker.repository.cloudera.com/cloudera/cdsw/ml-runtime-workbench-python3.9-standard:2022.11.2-b2",
       "python2": "docker.repository.cloudera.com/cloudera/cdsw/ml-runtime-workbench-python3.9-standard:2022.11.2-b2",
       "r": "docker.repository.cloudera.com/cloudera/cdsw/ml-runtime-workbench-r4.1-standard:2022.11.2-b2",
       "scala": "docker.repository.cloudera.com/cloudera/cdsw/ml-runtime-workbench-scala2.11-standard:2022.11.2-b2",
       "default": "docker.repository.cloudera.com/cloudera/cdsw/ml-runtime-workbench-python3.9-standard:2023.05.2-b7"
      }
   

4. Modifying contents of the mapping

   The mapping in the <home-directory>/.cmlutils/legacy_engine_runtime_constants.json file can be edited to customise the runtimes mapping as required.

Legacy engine to engine migration (Applicable only for Private Cloud)

Projects using the legacy engine can be migrated to engine based projects by following below process

1. Create an empty <home-directory>/.cmlutils/legacy_engine_runtime_constants.json i.e contents of the json should be {}
2. Run the export command
3. Run the import command

Disclaimer

Public Cloud

Legacy Engines are deprecated since June 2021 and are removed in recent releases. Starting with version 2.0.38, on new workspaces Legacy Engines are disabled by default and no Legacy Engine image is registered in the workspace. Cloudera recommends using ML Runtimes for all new projects, and urges customers to migrate existing Engine-based projects to ML Runtimes. Refer this

Runtimes using EOS Kernels

Python 3.6,3.7 are EOS by the community and Cloudera is not maintaining the kernels beyond that point.

• If you are still willing to use these runtimes, you can build a self maintained custom PBJ Runtime. https://docs.cloudera.com/machine-learning/1.5.1/runtimes/topics/ml-pbj-workbench-dockerfile.html

• Other option is to use the ActiveState Platform to build a custom ML Runtime that you can register directly in CML. https://blog.cloudera.com/how-to-ensure-supply-chain-security-for-ai-applications/