AWS/Sagemaker dynamic pipelines (#4232)

* First working prototype - no schedules, no isolated steps, duplicated code

* Added support to launch dynamic steps

* Resolved code duplication and reworked metadata extraction

* Fix use of processing jobs

* Fix docstrings

* Add metadata for sync dynamic pipeline runs

* Code review improvements

* Add docs

Author: stefannica

docs/book/component-guide/orchestrators/sagemaker.md

@@ -165,7 +165,7 @@ Create a custom policy for the execution role that SageMaker will assume:
 
 ### Client Permissions (for pipeline submission)
 
-Create a custom policy for the client/user submitting pipelines:
+Create a custom policy for the client/user submitting pipelines and training/processing jobs:
 
 ```json
 {
@@ -183,6 +183,12 @@ Create a custom policy for the client/user submitting pipelines:
         "sagemaker:ListPipelineExecutionSteps",
         "sagemaker:UpdatePipeline",
         "sagemaker:DeletePipeline"
+        "sagemaker:CreateProcessingJob",
+        "sagemaker:DescribeProcessingJob",
+        "sagemaker:StopProcessingJob",
+        "sagemaker:CreateTrainingJob",
+        "sagemaker:DescribeTrainingJob",
+        "sagemaker:StopTrainingJob"        
       ],
       "Resource": "*"
     },
@@ -206,7 +212,7 @@ Replace `ACCOUNT-ID` and `EXECUTION-ROLE-NAME` with your actual values.
 ZenML will build a Docker image called `<CONTAINER_REGISTRY_URI>/zenml:<PIPELINE_NAME>` which includes your code and use it to run your pipeline steps in Sagemaker. Check out [this page](https://docs.zenml.io/how-to/customize-docker-builds/) if you want to learn more about how ZenML builds these images and how you can customize them.
 {% endhint %}
 
-You can now run any ZenML pipeline using the Sagemaker orchestrator:
+You can now run any ZenML static pipeline or dynamic pipeline using the Sagemaker orchestrator:
 
 ```shell
 python run.py
@@ -237,6 +243,8 @@ Once the Studio UI has launched, click on the 'Pipeline' button on the left side
 
 ![Sagemaker Studio Pipelines](../../.gitbook/assets/sagemakerUI.png)
 
+If you are running dynamic pipelines, you can access the training/processing jobs in the SageMaker UI by clicking on the 'Jobs' button on the left side. From there you can view the jobs that have been launched via ZenML. A training job will be created for each dynamic pipeline run and for each step in the dynamic pipeline marked to run as an isolated step.
+
 ### Debugging SageMaker Pipelines
 
 If your SageMaker pipeline encounters an error before the first ZenML step starts, the ZenML run will not appear in the ZenML dashboard. In such cases, use the [SageMaker UI](sagemaker.md#sagemaker-ui) to review the error message and logs. Here's how:
@@ -470,6 +478,10 @@ Note that if you wish to use this orchestrator to run steps on a GPU, you will n
 
 ### Scheduling Pipelines
 
+{% hint style="warning" %}
+The SageMaker orchestrator does not support scheduling for dynamic pipelines yet.
+{% endhint %}
+
 The SageMaker orchestrator supports running pipelines on a schedule using SageMaker's native scheduling capabilities. You can configure schedules in three ways:
 
 * Using a cron expression