Improve CLI Tables and IDs handling (#4241)

Author: safoinme

docker/zenml-dev.Dockerfile

@@ -43,10 +43,11 @@ ENV PATH="$VIRTUAL_ENV/bin:$PATH"
 
 COPY README.md pyproject.toml ./
 
-# We first copy the __init__.py file to allow pip install-ing the Python
+# We first copy the __init__.py files to allow pip install-ing the Python
 # dependencies as a separate cache layer. This way, we can avoid re-installing
 # the dependencies when the source code changes but the dependencies don't.
 COPY src/zenml/__init__.py ./src/zenml/
+COPY src/zenml_cli/__init__.py ./src/zenml_cli/
 
 # Run pip install before copying the source files to install dependencies in
 # the virtual environment. Also create a requirements.txt file to keep track of

docker/zenml-server-dev.Dockerfile

@@ -76,10 +76,11 @@ ENV PATH="$VIRTUAL_ENV/bin:$PATH"
 
 COPY --chown=$USERNAME:$USER_GID  README.md pyproject.toml ./
 
-# We first copy the __init__.py file to allow pip install-ing the Python
+# We first copy the __init__.py files to allow pip install-ing the Python
 # dependencies as a separate cache layer. This way, we can avoid re-installing
 # the dependencies when the source code changes but the dependencies don't.
 COPY --chown=$USERNAME:$USER_GID src/zenml/__init__.py ./src/zenml/
+COPY --chown=$USERNAME:$USER_GID src/zenml_cli/__init__.py ./src/zenml_cli/
 
 # Run pip install before copying the source files to install dependencies in
 # the virtual environment. Also create a requirements.txt file to keep track of

docs/book/reference/environment-variables.md

@@ -129,6 +129,28 @@ To set the path to the global config file, used by ZenML to manage and store the
 export ZENML_CONFIG_PATH=/path/to/somewhere
 ```
 
+## CLI output formatting
+
+### Default output format
+
+Set the default output format for all CLI list commands:
+
+```bash
+export ZENML_DEFAULT_OUTPUT=json
+```
+
+Choose from `table` (default), `json`, `yaml`, `csv`, or `tsv`. This applies to commands like `zenml stack list`, `zenml pipeline list`, etc.
+
+### Terminal width override
+
+Override the automatic terminal width detection for table rendering:
+
+```bash
+export ZENML_CLI_COLUMN_WIDTH=120
+```
+
+This is useful when running ZenML in CI/CD environments or when you want to control table formatting regardless of your terminal size.
+
 ## Server configuration
 
 For more information on server configuration, see the [ZenML Server documentation](../getting-started/deploying-zenml/deploy-with-docker.md#zenml-server-configuration-options) for more, especially the section entitled "ZenML server configuration options".

docs/book/reference/global-settings.md

@@ -47,6 +47,10 @@ Using the default local database.
 ┗━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━┛
 ```
 
+{% hint style="info" %}
+The output can be customized with an `--output` (json, yaml, csv, tsv, table) option and a `--columns` selection. See [environment variables](environment-variables.md#cli-output-formatting) for more details.
+{% endhint %}
+
 The following is an example of the layout of the global config directory immediately after initialization:
 
 ```

docs/book/user-guide/best-practices/quick-wins.md

@@ -25,6 +25,7 @@ micro-setup (under 5 minutes) and any tips or gotchas to anticipate.
 | [Model Control Plane](#id-12-register-models-in-the-model-control-plane) | Track models and their lifecycle | Central hub for model lineage and governance |
 | [Parent Docker images](#id-13-create-a-parent-docker-image-for-faster-builds) | Pre-configure your dependencies in a base image | Faster builds and consistent environments |
 | [ZenML docs via MCP](#id-14-enable-ide-ai-zenml-docs-via-mcp-server) | Connect your IDE assistant to live ZenML docs | Faster, grounded answers and doc lookups while coding |
+| [Export CLI data](#id-15-export-cli-data-in-multiple-formats) | Get machine-readable output from list commands | Perfect for scripting, automation, and data analysis |
 
 
 ## 1 Log rich metadata on every run
@@ -854,3 +855,61 @@ Using the zenmldocs MCP server, show me how to register an MLflow experiment tra
 - For bleeding-edge features on develop, consult the repo or develop docs directly
 
 Learn more: [Access ZenML documentation via llms.txt and MCP](https://docs.zenml.io/reference/llms-txt)
+
+## 15 Export CLI data in multiple formats
+
+All `zenml list` commands support multiple output formats for scripting, CI/CD integration, and data analysis.
+
+```bash
+# Get stack data as JSON for processing with jq
+zenml stack list --output=json | jq '.items[] | select(.name=="production")'
+
+# Export pipeline runs to CSV for analysis
+zenml pipeline runs list --output=csv > pipeline_runs.csv
+
+# Get deployment info as YAML for configuration management
+zenml deployment list --output=yaml
+
+# Filter columns to see only what you need
+zenml stack list --columns=id,name,orchestrator
+
+# Combine filtering with custom output formats
+zenml pipeline list --columns=id,name,num_runs --output=json
+```
+
+**Available formats**
+- **json** - Structured data with pagination info, perfect for programmatic processing
+- **yaml** - Human-readable structured format, great for configuration
+- **csv** - Comma-separated values for spreadsheets and data analysis
+- **tsv** - Tab-separated values for simpler parsing
+- **table** (default) - Formatted tables with colors and alignment
+
+**Key features**
+- **Column filtering** - Use `--columns` to show only the fields you need
+- **Scriptable** - Combine with tools like `jq`, `grep`, `awk` for powerful automation
+- **Environment control** - Set `ZENML_DEFAULT_OUTPUT` to change the default format
+- **Width control** - Override terminal width with `ZENML_CLI_COLUMN_WIDTH` for consistent formatting
+
+**Best practices**
+- Use JSON format for robust parsing in scripts (includes pagination metadata)
+- Use CSV/TSV for importing into spreadsheet tools or databases
+- Use `--columns` to reduce noise and focus on relevant data
+- Set default formats via environment variables in CI/CD environments
+
+**Example automation script**
+```bash
+#!/bin/bash
+# Export all production stacks to a report
+
+export ZENML_DEFAULT_OUTPUT=json
+
+# Get all stacks and filter for production
+zenml stack list | jq '.items[] | select(.name | contains("prod"))' > prod_stacks.json
+
+# Generate a summary CSV
+zenml stack list --output=csv --columns=name,orchestrator,artifact_store > stack_summary.csv
+
+echo "Reports generated: prod_stacks.json and stack_summary.csv"
+```
+
+Learn more: [Environment Variables](https://docs.zenml.io/reference/environment-variables#cli-output-formatting)

docs/book/user-guide/production-guide/understand-stacks.md

@@ -49,6 +49,10 @@ Stack 'default' with id '...' is owned by user default and is 'private'.
 ...
 ```
 
+{% hint style="info" %}
+You can customize the output using `--columns` to show specific fields or `--output` to change the format (json, yaml, csv, tsv). Learn more in the [Quick Wins guide](../best-practices/quick-wins.md#id-15-export-cli-data-in-multiple-formats).
+{% endhint %}
+
 {% hint style="info" %}
 As you can see a stack can be **active** on your **client**. This simply means that any pipeline you run will be using the **active stack** as its environment.
 {% endhint %}

pyproject.toml

@@ -2,6 +2,9 @@
 requires = ["uv_build >= 0.8.17, <0.9.0"]
 build-backend = "uv_build"
 
+[tool.uv.build-backend]
+module-name = ["zenml", "zenml_cli"]
+
 [project]
 name = "zenml"
 version = "0.92.0"
@@ -51,7 +54,7 @@ Repository = "https://github.com/zenml-io/zenml"
 Issues = "https://github.com/zenml-io/zenml/issues"
 
 [project.scripts]
-zenml = "zenml.cli.cli:cli"
+zenml = "zenml_cli:cli"
 
 [project.optional-dependencies]
 local = [

src/zenml/cli/__init__.py

@@ -225,6 +225,45 @@
 This syntax can also be combined to create more complex filters using the `or`
 and `and` keywords.
 
+Output formats
+--------------
+
+All ``list`` commands support multiple output formats for scripting,
+CI/CD integration, and data analysis.
+
+Use the ``--output`` (or ``-o``) option to specify the format:
+
+```bash
+# Get stack data as JSON for processing with jq
+zenml stack list --output=json | jq '.items[] | select(.name=="production")'
+
+# Export pipeline runs to CSV for analysis
+zenml pipeline runs list --output=csv > pipeline_runs.csv
+
+# Get deployment info as YAML for configuration management
+zenml deployment list --output=yaml
+
+# Filter columns to see only what you need
+zenml stack list --columns=id,name,orchestrator
+
+# Combine filtering with custom output formats
+zenml pipeline list --columns=id,name --output=json
+```
+
+Available formats:
+
+- **json** - Structured data with pagination info, ideal for programmatic use
+- **yaml** - Human-readable structured format, great for configuration
+- **csv** - Comma-separated values for spreadsheets and data analysis
+- **tsv** - Tab-separated values for simpler parsing
+- **table** (default) - Formatted tables with colors and alignment
+
+You can also control the default output format and table width using
+environment variables:
+
+- ``ZENML_DEFAULT_OUTPUT`` - Set the default output format (e.g., ``json``)
+- ``ZENML_CLI_COLUMN_WIDTH`` - Override terminal width for consistent formatting
+
 Artifact Stores
 ---------------
 

src/zenml/cli/annotator.py

@@ -73,9 +73,8 @@ def dataset_list(annotator: "BaseAnnotator") -> None:
         if not dataset_names:
             cli_utils.warning("No datasets found.")
             return
-        cli_utils.print_list_items(
-            list_items=dataset_names,
-            column_title="DATASETS",
+        cli_utils.print_table(
+            [{"DATASETS": name} for name in sorted(dataset_names)]
         )
 
     @dataset.command("stats")

src/zenml/cli/artifact.py

@@ -19,7 +19,9 @@
 
 from zenml.cli import utils as cli_utils
 from zenml.cli.cli import TagGroup, cli
+from zenml.cli.utils import OutputFormat, list_options
 from zenml.client import Client
+from zenml.console import console
 from zenml.enums import CliCategories
 from zenml.logger import get_logger
 from zenml.models import ArtifactFilter, ArtifactVersionFilter
@@ -35,25 +37,27 @@ def artifact() -> None:
     """Commands for interacting with artifacts."""
 
 
-@cli_utils.list_options(ArtifactFilter)
 @artifact.command("list", help="List all artifacts.")
-def list_artifacts(**kwargs: Any) -> None:
+@list_options(
+    ArtifactFilter,
+    default_columns=["id", "name", "latest_version_name", "tags"],
+)
+def list_artifacts(
+    columns: str, output_format: OutputFormat, **kwargs: Any
+) -> None:
     """List all artifacts.
 
     Args:
+        columns: Columns to display in output.
+        output_format: Format for output (table/json/yaml/csv/tsv).
         **kwargs: Keyword arguments to filter artifacts by.
     """
-    artifacts = Client().list_artifacts(**kwargs)
-
-    if not artifacts:
-        cli_utils.declare("No artifacts found.")
-        return
+    with console.status("Listing artifacts...\n"):
+        artifacts = Client().list_artifacts(**kwargs)
 
-    to_print = []
-    for artifact in artifacts:
-        to_print.append(_artifact_to_print(artifact))
-
-    cli_utils.print_table(to_print)
+    cli_utils.print_page(
+        artifacts, columns, output_format, empty_message="No artifacts found."
+    )
 
 
 @artifact.command("update", help="Update an artifact.")
@@ -115,25 +119,30 @@ def version() -> None:
     """Commands for interacting with artifact versions."""
 
 
-@cli_utils.list_options(ArtifactVersionFilter)
 @version.command("list", help="List all artifact versions.")
-def list_artifact_versions(**kwargs: Any) -> None:
+@list_options(
+    ArtifactVersionFilter,
+    default_columns=["id", "artifact", "version", "type"],
+)
+def list_artifact_versions(
+    columns: str, output_format: OutputFormat, **kwargs: Any
+) -> None:
     """List all artifact versions.
 
     Args:
+        columns: Columns to display in output.
+        output_format: Format for output (table/json/yaml/csv/tsv).
         **kwargs: Keyword arguments to filter artifact versions by.
     """
-    artifact_versions = Client().list_artifact_versions(**kwargs)
-
-    if not artifact_versions:
-        cli_utils.declare("No artifact versions found.")
-        return
-
-    to_print = []
-    for artifact_version in artifact_versions:
-        to_print.append(_artifact_version_to_print(artifact_version))
-
-    cli_utils.print_table(to_print)
+    with console.status("Listing artifact versions...\n"):
+        artifact_versions = Client().list_artifact_versions(**kwargs)
+
+    cli_utils.print_page(
+        artifact_versions,
+        columns,
+        output_format,
+        empty_message="No artifact versions found.",
+    )
 
 
 @version.command("describe", help="Show details about an artifact version.")