From 6a56ab51971c1c19010f9c49326e375712a21290 Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Fri, 31 Oct 2025 15:16:56 +0100 Subject: [PATCH 01/18] docs(design): service provider On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/_category_.yml | 1 + docs/about/design/service-provider-design.md | 153 +++++++++++++++++++ 2 files changed, 154 insertions(+) create mode 100644 docs/about/design/_category_.yml create mode 100644 docs/about/design/service-provider-design.md diff --git a/docs/about/design/_category_.yml b/docs/about/design/_category_.yml new file mode 100644 index 0000000..7c52272 --- /dev/null +++ b/docs/about/design/_category_.yml @@ -0,0 +1 @@ +label: Design diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md new file mode 100644 index 0000000..42a3a05 --- /dev/null +++ b/docs/about/design/service-provider-design.md @@ -0,0 +1,153 @@ +# Service Provider Design + +## Goals + +- Define clear terminology around `ServiceProvider` in the OpenMCP space +- Define `ServiceProvider` scope: responsibilities and boundaries of a `ServiceProvider` +- Define a `ServiceProvider` model that implements the higher level `API`/`Run` platform concept (to allow flexible deployment models, e.g. with `ClusterProvider` kcp) +- Define `ServiceProvider` contract to implement `ServiceProvider` as a loosely coupled component in the openMCP context +- Define how a `ServiceProvider` can be validated + +## Non-Goals + +tbd + +## Object Model + +```mermaid +graph TD + %% Onboarding Cluster + subgraph OnboardingCluster/API + SC[ServiceConfig] + end + + %% Platform Cluster + subgraph PlatformCluster/RUN + SPO[service-provider-operator] + SP[ServiceProvider] + SPC[ServiceProviderConfig] + end + + %% MCP Cluster + subgraph MCPCluster/RUN + DS[DomainService] + DSAPI[DomainServiceAPI] + end + + %% WorkloadCluster + subgraph WorkloadCluster/RUN + SDS[SharedDomainService] + end + + %% edges + SP -->|installs/reconciles|SC + SP -->|uses|SPC + SP -->|creates/updates/deletes|DS + SP -->|creates/updates/deletes|SDS + DS -->|installs/reconciles|DSAPI + SDS --->|reconciles/XOR|DSAPI + SPO-->|installs/reconciles|SP +``` + +Open Points: + +- Does the `openmcp-operator` manage `ServiceProviders` or do we introduce a new operator for `ServiceProviders`? Benefits of a new component could be clear separation of concerns. The `openmcp-operator` already does a lot and we don't want the next `control-plane-operator`. +- In the above model the `OnboardingCluster` is a continuous `API` cluster. We might want to provision dedicated or shared tenant `API` servers (e.g. with `ClusterProvider` kcp) based on some kind of component discovery that lets the tenant pick its feature/component set. This way the `OnboardingCluster` is only used to onboard new tenants. And we don't run into CRD management hell/bottlenecks. +- Another thought regarding the `OnboardingCluster`. If we introduce tenant `API` clusters, they could be used to create MCPs. This again implies that instead of having the `OnboardingCluster` create `MCPs`, we might want to have the `OnboardingCluster` create `Tenants` as the entry point for users -> start with an identity object like `Tenant` or `Account` instead of a usage artifact like `MCP`. + +TODO: + +- Illustrate different deployment models with `Run`/`API` concept +- Visually distinguish between `Run` and `API` artifacts + +## Terminology + +Defines the objects of the [object model](#object-model) + +- `ServiceProvider` provides a service in tenant space +- `PlatformService` provides a service in platform space +- `Run` clusters support scheduling workloads. A `Run` cluster may or may not also serve as `API` cluster. +- `API` clusters serve APIs but do not support scheduling workload (note that `API`/`Run` is a higher level platform concept) +- `OnboardingCluster` is part of the platform domain and the config/setup part from a tenant perspective. It serves the `API` of a `ServiceProvider` +- `MCPCluster` is part of the tenant domain and the application/functional part from a tenant perspective. It may or may not run the `Run` of a `ServiceProvider` +- `PlatformCluster` is part of the platform domain and a black box from a tenant perspective. It may or may not run the `Run` of a `ServiceProvider` +- A `ServiceConfig` defines the service provisioning in terms of the `DomainService` `API` and `Run` where e.g. Crossplane could be provisioned for a tenant by installing the `API` on the tenant MCP but the `Run` on a shared worker pool (`WorkloadCluster`) (clarify tenant IAM). A tenant can use this mechanism to decide how to consume a service. +- A `ServiceProviderConfig` defines the config parts that are used in reconcile run, e.g. to define tenant boundaries + +## Boundaries + +- A `PlatformService` (e.g. `service-provider-operator`) watches platform `API` clusters, e.g. the `OnboardingCluster` and acts on platform `Run` clusters, e.g. itself or shared `WorkloadClusters`. It does not act on tenant clusters, e.g. MCPs +- A `ServiceProvider` watches tenant `API` clusters, e.g. the `OnboardingCluster` and acts on `Run` clusters, e.g. MCPs. + +tbc platform space vs tenant space + +## Lifecycle + +- A `PlatformService` is installed by a platform team and/or bootstrapping mechanism (out of scope) +- A `ServiceProvider` is installed by creating ServiceProvider objects, the `service-provider-operator` manages the lifecycle of `ServiceProviders`... advantages disadvantages + +## Validation + +A `ServiceProvider` is considered healthy if both its `API` and `Run` part have been successfully synced and are ready for consumption. + +The following validation flow validates that a `ServiceProvider` is working as expected: + +0. SETUP: Create test environment by installing any `ServiceProvider` prerequisite: a) k8s cluster, e.g. kind, b) install `service-provider-operator` -> wait for operator to be available +1. ASSESS: Request `ServiceProvider` -> wait for `API` and `Run` components to be `synced` and `ready` +2. ASSESS: Consume `API` to provision `DomainService` -> wait for DomainService to be `synced` and `ready` +3. ASSESS: (optional) Consume `DomainServiceAPI` depending on the provider/domain context this may or may not be required +4. ASSESS: Delete `ServiceProvider` -> wait for `API`, `Run`, `ServiceProvider` to be successfully removed +5. TEARDOWN: Delete test environment components + +## Runtime + +What is a runtime? A runtime is a collection of abstractions and contracts that provides an environment in which user-defined logic is executed. + +The service provider runtime is built on top of controller-runtime and provides a service provider specific reconciliation loop. + +It provides: + +- client abstractions (in xp external clients, in openmcp e.g. reuse common juggler reconcilers like flux?) +- lifecycle management abstractions of `ServiceProviderAPI` objects (the reconcile loop) +- platform specific features (in xp e.g. late initialize, external-name and pause annotations), enables us to implement platform features for all service providers (a `ServiceProvider` only needs to update their runtime dependency) +- handling of cross-cutting concerns like event recording, logging, metrics, rate limits + +The following overview illustrates the layers in a simplified way: + +| Layer | Description | +| :--- | :--- | +| Service Provider | defines `ServiceProviderAPI` and implements service-provider-runtime operations | +| service-provider-runtime | defines ServiceProvider reconciliation semantics | +| controller-runtime | defines generic reconciliation semantics | +| Kubernetes API machinery | k8s essentials | +| Go runtime / OS kernel | process/thread execution, memory management | + +### Execution Model + +Here we define what a run/reconcile cycle means, e.g. observe followed by an orchestration of actions like create, update, delete. + +This may include special domain semantics similar to `ManagementPolicies` or the `pause` state/mechanism in Crossplane. + +### Abstractions and Contracts + +Here we define the core interfaces that a consumer (`ServiceProvider` developer) has to implement, e.g. in Crossplane `ExternalConnector` creates `ExternalClient` which implements CRUD operations with `ExternalObservation`, `ExternalCreation`, etc. `Managed` interface defines what makes a k8s object a managed Crossplane resource, e.g. by referencing a `ProviderConfig`, specifying `ManagementPolicies`, `ConnectionSecrets`, etc. + +### Observability + +Logging, metrics, traces? + +## Domain + +The actual domain layer of a `ServiceProvider` (layer on top of the [runtime](#runtime)). The foundation to build a `ServiceProvider` template. + +### RBAC + +What permissions does a service provider need... + +## Service Provider Manager + +The component that manages the lifecyclee of `ServiceProviders` and provides service discovery to platform `API` clusters, e.g. `OnboardingCluster`. + +candidates e.g. `openmcp-operator` or `service-provider-operator` + +out of scope? From 72f9ab08b7e5283d4ff6e9d4fbf2cce33e41563f Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Mon, 3 Nov 2025 11:44:26 +0100 Subject: [PATCH 02/18] docs(design): service provider domain On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 28 +++++++++++++++----- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index 42a3a05..e3f86c5 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -7,6 +7,7 @@ - Define a `ServiceProvider` model that implements the higher level `API`/`Run` platform concept (to allow flexible deployment models, e.g. with `ClusterProvider` kcp) - Define `ServiceProvider` contract to implement `ServiceProvider` as a loosely coupled component in the openMCP context - Define how a `ServiceProvider` can be validated +- (MCP) v1 learnings have been addressed ## Non-Goals @@ -101,7 +102,7 @@ The following validation flow validates that a `ServiceProvider` is working as e ## Runtime -What is a runtime? A runtime is a collection of abstractions and contracts that provides an environment in which user-defined logic is executed. +A runtime is a collection of abstractions and contracts that provides an environment in which user-defined logic is executed. The service provider runtime is built on top of controller-runtime and provides a service provider specific reconciliation loop. @@ -132,17 +133,18 @@ This may include special domain semantics similar to `ManagementPolicies` or the Here we define the core interfaces that a consumer (`ServiceProvider` developer) has to implement, e.g. in Crossplane `ExternalConnector` creates `ExternalClient` which implements CRUD operations with `ExternalObservation`, `ExternalCreation`, etc. `Managed` interface defines what makes a k8s object a managed Crossplane resource, e.g. by referencing a `ProviderConfig`, specifying `ManagementPolicies`, `ConnectionSecrets`, etc. -### Observability - -Logging, metrics, traces? - ## Domain The actual domain layer of a `ServiceProvider` (layer on top of the [runtime](#runtime)). The foundation to build a `ServiceProvider` template. -### RBAC +A `ServiceProvider` has the following responsibilities: + +- Manage the lifecycle of the `API` and `Run` of a `DomainService`. +- Allow multiple `APIClusters` to target the same `RunCluster`, e.g. the Crossplane managed resources on `MCP` A and `MCP` B are reconciled by the same Crossplane installation on a shared `WorkloadCluster`. -What permissions does a service provider need... +## Template / Builder + +Do we want a CLI like kubebuilder or a template like crossplane provider template? ## Service Provider Manager @@ -151,3 +153,15 @@ The component that manages the lifecyclee of `ServiceProviders` and provides ser candidates e.g. `openmcp-operator` or `service-provider-operator` out of scope? + +## Ideas + +- `SoftDelete` platform concept. A `managed` service can transition to a `unmanaged` service by soft deleting its corresponding `ServiceProviderAPI` or the `ServiceProvider` entirely without losing the `DomainService`. This way a tenant could offboard itself partially or entirely from the platform without losing the provisioned infrastructure. This obviously depends on the ownership model of the infrastructure. + +## References + +Projects in the same problem space: + +- [Crossplane](https://www.crossplane.io/) +- [kube-bind](https://github.com/kube-bind/kube-bind) +- [multi-cluster-runtime](https://github.com/kubernetes-sigs/multicluster-runtime) From ef1b1515721bf76e5ab907230032e2be2a227d46 Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Mon, 3 Nov 2025 11:52:20 +0100 Subject: [PATCH 03/18] docs(design): service provider domain On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index e3f86c5..5f62ac5 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -137,7 +137,7 @@ Here we define the core interfaces that a consumer (`ServiceProvider` developer) The actual domain layer of a `ServiceProvider` (layer on top of the [runtime](#runtime)). The foundation to build a `ServiceProvider` template. -A `ServiceProvider` has the following responsibilities: +A `ServiceProvider` defines how a `DomainService` can be consumed by a tenant. It has the following responsibilities: - Manage the lifecycle of the `API` and `Run` of a `DomainService`. - Allow multiple `APIClusters` to target the same `RunCluster`, e.g. the Crossplane managed resources on `MCP` A and `MCP` B are reconciled by the same Crossplane installation on a shared `WorkloadCluster`. From 89ec1849192c6a4f81f3a4f58fe30d4774e19140 Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Mon, 3 Nov 2025 13:12:48 +0100 Subject: [PATCH 04/18] docs(serviceprovider): multicluster-runtime facade On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index 5f62ac5..d9e63d1 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -119,10 +119,12 @@ The following overview illustrates the layers in a simplified way: | :--- | :--- | | Service Provider | defines `ServiceProviderAPI` and implements service-provider-runtime operations | | service-provider-runtime | defines ServiceProvider reconciliation semantics | -| controller-runtime | defines generic reconciliation semantics | +| multicluster/controller-runtime | defines generic reconciliation semantics | | Kubernetes API machinery | k8s essentials | | Go runtime / OS kernel | process/thread execution, memory management | +Multi-cluster functionality will most likely be part of `service-provider-runtime`, e.g. a facade-like layer on top of `multicluster-runtime` to enable service deployment on shared `WorkloadCluster`. + ### Execution Model Here we define what a run/reconcile cycle means, e.g. observe followed by an orchestration of actions like create, update, delete. @@ -140,7 +142,9 @@ The actual domain layer of a `ServiceProvider` (layer on top of the [runtime](#r A `ServiceProvider` defines how a `DomainService` can be consumed by a tenant. It has the following responsibilities: - Manage the lifecycle of the `API` and `Run` of a `DomainService`. -- Allow multiple `APIClusters` to target the same `RunCluster`, e.g. the Crossplane managed resources on `MCP` A and `MCP` B are reconciled by the same Crossplane installation on a shared `WorkloadCluster`. +- Provide its platform facing `API` -> `ServiceProviderConfig` +- Provide its tenant facing `API` -> `ServiceConfig` +- Allow multiple `APIClusters` to target the same `RunCluster`, e.g. the Crossplane managed resources on `MCP` A and `MCP` B are reconciled by the same Crossplane installation on a shared `WorkloadCluster` -> we can think of this as a `multicluster-runtime` facade that is part of [service-provider-runtime](#runtime). ## Template / Builder @@ -160,8 +164,8 @@ out of scope? ## References -Projects in the same problem space: +Projects with similar concepts: - [Crossplane](https://www.crossplane.io/) - [kube-bind](https://github.com/kube-bind/kube-bind) -- [multi-cluster-runtime](https://github.com/kubernetes-sigs/multicluster-runtime) +- [multicluster-runtime](https://github.com/kubernetes-sigs/multicluster-runtime) From a96b76eb3af450ad80e8d83a32b7ba0e65b3a59f Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Mon, 3 Nov 2025 13:37:39 +0100 Subject: [PATCH 05/18] docs(serviceprovider): multicluster-runtime facade On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index d9e63d1..9a8f0de 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -125,6 +125,17 @@ The following overview illustrates the layers in a simplified way: Multi-cluster functionality will most likely be part of `service-provider-runtime`, e.g. a facade-like layer on top of `multicluster-runtime` to enable service deployment on shared `WorkloadCluster`. +```mermaid +graph TD + SPC[service-provider-controller] + SPR[service-provider-runtime] + MCR[multicluster-runtime] + + %% edges + SPC -->|imports|SPR + SPR -->|imports/abstracts|MCR +``` + ### Execution Model Here we define what a run/reconcile cycle means, e.g. observe followed by an orchestration of actions like create, update, delete. From b043a74e1fd48eaaf98d56ffb5b8553423a8c54a Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Mon, 3 Nov 2025 14:55:38 +0100 Subject: [PATCH 06/18] docs(serviceprovider): multicluster-runtime vs juggler On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 80 +++++++++++++++++++- 1 file changed, 79 insertions(+), 1 deletion(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index 9a8f0de..f87b13b 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -123,7 +123,7 @@ The following overview illustrates the layers in a simplified way: | Kubernetes API machinery | k8s essentials | | Go runtime / OS kernel | process/thread execution, memory management | -Multi-cluster functionality will most likely be part of `service-provider-runtime`, e.g. a facade-like layer on top of `multicluster-runtime` to enable service deployment on shared `WorkloadCluster`. +Multi-cluster functionality will most likely be part of `service-provider-runtime`, e.g. a layer on top of `multicluster-runtime` or an `object-syncer/juggler` to enable service deployment on shared `WorkloadCluster`. ```mermaid graph TD @@ -142,6 +142,84 @@ Here we define what a run/reconcile cycle means, e.g. observe followed by an orc This may include special domain semantics similar to `ManagementPolicies` or the `pause` state/mechanism in Crossplane. +The following two diagrams illustrate the `juggler` vs `multicluster-runtime` versions. + +"Multicluster-runtime" facade version (not feasable because essentially this means replacing controller-runtime with multicluster-runtime in domain controllers): + +```mermaid +graph TD + %% WorkloadCluster + subgraph WorkloadCluster/RUN + DS[DomainService/RUN] + subgraph ServiceProviderInstance + SPR[service-provider-runtime] + end + end + + %% MCPClusterA + subgraph MCPClusterA/API + DSAA[DomainServiceAPI] + end + + %% MCPClusterB + subgraph MCPClusterB/API + DSAB[DomainServiceAPI] + end + + %% MCPClusterC + subgraph MCPClusterC/API + DSAC[DomainServiceAPI] + end + + %% edges + DS ---|forwardRequests/syncStatus| SPR + SPR -->|reconciles|DSAA + SPR -->|reconciles|DSAB + SPR -->|reconciles|DSAC +``` + +Object syncer / juggler version: + +```mermaid +graph TD + %% WorkloadCluster + subgraph WorkloadCluster/RUN + DS[DomainService/RUN] + DSAACopy[DomainServiceAPICopyA] + DSABCopy[DomainServiceAPICopyB] + DSACCopy[DomainServiceAPICopyC] + subgraph ServiceProviderInstance + SPR[service-provider-runtime] + end + end + + %% MCPClusterA + subgraph MCPClusterA/API + DSAA[DomainServiceAPI] + end + + %% MCPClusterB + subgraph MCPClusterB/API + DSAB[DomainServiceAPI] + end + + %% MCPClusterC + subgraph MCPClusterC/API + DSAC[DomainServiceAPI] + end + + %% edges + DS -->|reconciles| DSAACopy + DS -->|reconciles| DSABCopy + DS -->|reconciles| DSACCopy + DSAACopy ---|create/sync| SPR + DSABCopy ---|create/sync| SPR + DSACCopy ---|create/sync| SPR + SPR -->|sync|DSAA + SPR -->|sync|DSAB + SPR -->|sync|DSAC +``` + ### Abstractions and Contracts Here we define the core interfaces that a consumer (`ServiceProvider` developer) has to implement, e.g. in Crossplane `ExternalConnector` creates `ExternalClient` which implements CRUD operations with `ExternalObservation`, `ExternalCreation`, etc. `Managed` interface defines what makes a k8s object a managed Crossplane resource, e.g. by referencing a `ProviderConfig`, specifying `ManagementPolicies`, `ConnectionSecrets`, etc. From 6d230728568131a990fd049341c8a8fd51c02fe7 Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Mon, 3 Nov 2025 18:40:26 +0100 Subject: [PATCH 07/18] docs(serviceprovider): separated in scope and out of scope sections On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 136 ++++++++----------- 1 file changed, 55 insertions(+), 81 deletions(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index f87b13b..2a25484 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -11,7 +11,7 @@ ## Non-Goals -tbd +- `ServiceProviders` does not need to deploy its `DomainService` on `WorkloadClusters`. For now a `DomainService` can be deployed on both `WorkloadCluster` or `MCPCluster`. ## Object Model @@ -19,12 +19,12 @@ tbd graph TD %% Onboarding Cluster subgraph OnboardingCluster/API - SC[ServiceConfig] + SAPI[ServiceAPI] end %% Platform Cluster subgraph PlatformCluster/RUN - SPO[service-provider-operator] + SPO[openMCP-operator] SP[ServiceProvider] SPC[ServiceProviderConfig] end @@ -33,6 +33,7 @@ graph TD subgraph MCPCluster/RUN DS[DomainService] DSAPI[DomainServiceAPI] + SDSObject[SharedDomainServiceObject] end %% WorkloadCluster @@ -41,51 +42,33 @@ graph TD end %% edges - SP -->|installs/reconciles|SC + SP -->|installs/reconciles|SAPI SP -->|uses|SPC SP -->|creates/updates/deletes|DS SP -->|creates/updates/deletes|SDS DS -->|installs/reconciles|DSAPI - SDS --->|reconciles/XOR|DSAPI + SDS --->|creates/updates/deletes|SDSObject SPO-->|installs/reconciles|SP ``` -Open Points: +- The [openmcp-operator](https://github.com/openmcp-project/openmcp-operator) manages the lifecyclee of `ServiceProviders`. +- A `ServiceProvider` provides a third party service in tenant space. It manages the lifecycle of a `DomainService` and provides a `ServiceAPI`. +- The `ServiceProviderConfig` defines the platform facing config that is used in a reconcile run, e.g. to implement multi-tenancy on the `PlatformCluster`. +- The `ServiceAPI` is the tenant facing set of CRDs of a `ServiceProvider`. +- A `DomainService` is a third party service that manages `DomainServiceAPI` objects. +- The `DomainServiceAPI` is the set of CRDs introduced by `DomainService`. +- A `SharedDomainService` is a `DomainService` that can be deployed on a `WorkloadCluster`. A `SharedDomainService` does not expose its `DomainServiceAPI` to end users. +- A `SharedDomainServiceObject` is a Kubernets object managed by the `SharedDomainService`. -- Does the `openmcp-operator` manage `ServiceProviders` or do we introduce a new operator for `ServiceProviders`? Benefits of a new component could be clear separation of concerns. The `openmcp-operator` already does a lot and we don't want the next `control-plane-operator`. -- In the above model the `OnboardingCluster` is a continuous `API` cluster. We might want to provision dedicated or shared tenant `API` servers (e.g. with `ClusterProvider` kcp) based on some kind of component discovery that lets the tenant pick its feature/component set. This way the `OnboardingCluster` is only used to onboard new tenants. And we don't run into CRD management hell/bottlenecks. -- Another thought regarding the `OnboardingCluster`. If we introduce tenant `API` clusters, they could be used to create MCPs. This again implies that instead of having the `OnboardingCluster` create `MCPs`, we might want to have the `OnboardingCluster` create `Tenants` as the entry point for users -> start with an identity object like `Tenant` or `Account` instead of a usage artifact like `MCP`. - -TODO: - -- Illustrate different deployment models with `Run`/`API` concept -- Visually distinguish between `Run` and `API` artifacts - -## Terminology - -Defines the objects of the [object model](#object-model) - -- `ServiceProvider` provides a service in tenant space -- `PlatformService` provides a service in platform space -- `Run` clusters support scheduling workloads. A `Run` cluster may or may not also serve as `API` cluster. -- `API` clusters serve APIs but do not support scheduling workload (note that `API`/`Run` is a higher level platform concept) -- `OnboardingCluster` is part of the platform domain and the config/setup part from a tenant perspective. It serves the `API` of a `ServiceProvider` -- `MCPCluster` is part of the tenant domain and the application/functional part from a tenant perspective. It may or may not run the `Run` of a `ServiceProvider` -- `PlatformCluster` is part of the platform domain and a black box from a tenant perspective. It may or may not run the `Run` of a `ServiceProvider` -- A `ServiceConfig` defines the service provisioning in terms of the `DomainService` `API` and `Run` where e.g. Crossplane could be provisioned for a tenant by installing the `API` on the tenant MCP but the `Run` on a shared worker pool (`WorkloadCluster`) (clarify tenant IAM). A tenant can use this mechanism to decide how to consume a service. -- A `ServiceProviderConfig` defines the config parts that are used in reconcile run, e.g. to define tenant boundaries - -## Boundaries - -- A `PlatformService` (e.g. `service-provider-operator`) watches platform `API` clusters, e.g. the `OnboardingCluster` and acts on platform `Run` clusters, e.g. itself or shared `WorkloadClusters`. It does not act on tenant clusters, e.g. MCPs -- A `ServiceProvider` watches tenant `API` clusters, e.g. the `OnboardingCluster` and acts on `Run` clusters, e.g. MCPs. +## Domain -tbc platform space vs tenant space +A `ServiceProvider` defines how a `DomainService` can be consumed by a tenant. It has the following responsibilities: -## Lifecycle +- Manage the lifecycle of a `DomainService` +- Define platform facing config -> `ServiceProviderConfig` +- Define a tenant facing `API` -> `ServiceConfig` -- A `PlatformService` is installed by a platform team and/or bootstrapping mechanism (out of scope) -- A `ServiceProvider` is installed by creating ServiceProvider objects, the `service-provider-operator` manages the lifecycle of `ServiceProviders`... advantages disadvantages +A `DomainService` has to be treated as an external system where a `ServiceProvider` continuously has to prevent any drift caused by tenant actions on both `MCP` and `OnboardingCluster`. ## Validation @@ -100,6 +83,10 @@ The following validation flow validates that a `ServiceProvider` is working as e 4. ASSESS: Delete `ServiceProvider` -> wait for `API`, `Run`, `ServiceProvider` to be successfully removed 5. TEARDOWN: Delete test environment components +## Template + +tbd. + ## Runtime A runtime is a collection of abstractions and contracts that provides an environment in which user-defined logic is executed. @@ -113,7 +100,7 @@ It provides: - platform specific features (in xp e.g. late initialize, external-name and pause annotations), enables us to implement platform features for all service providers (a `ServiceProvider` only needs to update their runtime dependency) - handling of cross-cutting concerns like event recording, logging, metrics, rate limits -The following overview illustrates the layers in a simplified way: +The following overview illustrates the layers of a `ServiceProvider` controller a simplified way: | Layer | Description | | :--- | :--- | @@ -121,9 +108,20 @@ The following overview illustrates the layers in a simplified way: | service-provider-runtime | defines ServiceProvider reconciliation semantics | | multicluster/controller-runtime | defines generic reconciliation semantics | | Kubernetes API machinery | k8s essentials | -| Go runtime / OS kernel | process/thread execution, memory management | -Multi-cluster functionality will most likely be part of `service-provider-runtime`, e.g. a layer on top of `multicluster-runtime` or an `object-syncer/juggler` to enable service deployment on shared `WorkloadCluster`. +### Abstractions and Contracts + +Here we define the core interfaces that a consumer (`ServiceProvider` developer) has to implement, e.g. in Crossplane `ExternalConnector` creates `ExternalClient` which implements CRUD operations with `ExternalObservation`, `ExternalCreation`, etc. `Managed` interface defines what makes a k8s object a managed Crossplane resource, e.g. by referencing a `ProviderConfig`, specifying `ManagementPolicies`, `ConnectionSecrets`, etc. + +## Out of Scope + +The remainder of this document contains topics that are out of scope for now. + +### Multicluster Execution Model + +Multi-cluster functionality for `ServiceProvider` is a design goal for future iterations and might get integrated into `service-provider-runtime`. This would enable service deployment on shared `WorkloadCluster`. The following two diagrams illustrate a `juggler` and `multicluster-runtime` version. + +#### Multicluster-runtime Facade ```mermaid graph TD @@ -136,15 +134,7 @@ graph TD SPR -->|imports/abstracts|MCR ``` -### Execution Model - -Here we define what a run/reconcile cycle means, e.g. observe followed by an orchestration of actions like create, update, delete. - -This may include special domain semantics similar to `ManagementPolicies` or the `pause` state/mechanism in Crossplane. - -The following two diagrams illustrate the `juggler` vs `multicluster-runtime` versions. - -"Multicluster-runtime" facade version (not feasable because essentially this means replacing controller-runtime with multicluster-runtime in domain controllers): +Not feasable because this essentially means replacing/reimplementing existing domain controllers based on controller-runtime with multicluster-runtime. ```mermaid graph TD @@ -178,7 +168,9 @@ graph TD SPR -->|reconciles|DSAC ``` -Object syncer / juggler version: +#### Object Syncer / Juggler + +Another approach would be to sync API objects between `API` and `RUN` clusters as a feature of service-provider-runtime. ```mermaid graph TD @@ -212,46 +204,28 @@ graph TD DS -->|reconciles| DSAACopy DS -->|reconciles| DSABCopy DS -->|reconciles| DSACCopy - DSAACopy ---|create/sync| SPR - DSABCopy ---|create/sync| SPR - DSACCopy ---|create/sync| SPR + DSAACopy ---|sync| SPR + DSABCopy ---|sync| SPR + DSACCopy ---|sync| SPR SPR -->|sync|DSAA SPR -->|sync|DSAB SPR -->|sync|DSAC ``` -### Abstractions and Contracts - -Here we define the core interfaces that a consumer (`ServiceProvider` developer) has to implement, e.g. in Crossplane `ExternalConnector` creates `ExternalClient` which implements CRUD operations with `ExternalObservation`, `ExternalCreation`, etc. `Managed` interface defines what makes a k8s object a managed Crossplane resource, e.g. by referencing a `ProviderConfig`, specifying `ManagementPolicies`, `ConnectionSecrets`, etc. - -## Domain - -The actual domain layer of a `ServiceProvider` (layer on top of the [runtime](#runtime)). The foundation to build a `ServiceProvider` template. - -A `ServiceProvider` defines how a `DomainService` can be consumed by a tenant. It has the following responsibilities: - -- Manage the lifecycle of the `API` and `Run` of a `DomainService`. -- Provide its platform facing `API` -> `ServiceProviderConfig` -- Provide its tenant facing `API` -> `ServiceConfig` -- Allow multiple `APIClusters` to target the same `RunCluster`, e.g. the Crossplane managed resources on `MCP` A and `MCP` B are reconciled by the same Crossplane installation on a shared `WorkloadCluster` -> we can think of this as a `multicluster-runtime` facade that is part of [service-provider-runtime](#runtime). +### Ideas -## Template / Builder - -Do we want a CLI like kubebuilder or a template like crossplane provider template? - -## Service Provider Manager - -The component that manages the lifecyclee of `ServiceProviders` and provides service discovery to platform `API` clusters, e.g. `OnboardingCluster`. - -candidates e.g. `openmcp-operator` or `service-provider-operator` - -out of scope? +- `SoftDelete` platform concept. A `managed` service can transition to a `unmanaged` service by soft deleting its corresponding `ServiceProviderAPI` or the `ServiceProvider` entirely without losing the `DomainService`. This way a tenant could offboard itself partially or entirely from the platform without losing the provisioned infrastructure. This obviously depends on the ownership model of the infrastructure. +- In the above model the `OnboardingCluster` is a continuous `API` cluster. We might want to provision dedicated or shared tenant `API` servers (e.g. with `ClusterProvider` kcp) based on some kind of component discovery that lets the tenant pick its feature/component set. This way the `OnboardingCluster` is only used to onboard new tenants. And we don't run into CRD management hell/bottlenecks. +- Another thought regarding the `OnboardingCluster`. If we introduce tenant `API` clusters, they could be used to create MCPs. This again implies that instead of having the `OnboardingCluster` create `MCPs`, we might want to have the `OnboardingCluster` create `Tenants` as the entry point for users -> start with an identity object like `Tenant` or `Account` instead of a usage artifact like `MCP`. +- Distinguish between `Run` and `API` artifacts on all platform layers +- Illustrate different deployment models with `Run`/`API` concept -## Ideas +### Terminology -- `SoftDelete` platform concept. A `managed` service can transition to a `unmanaged` service by soft deleting its corresponding `ServiceProviderAPI` or the `ServiceProvider` entirely without losing the `DomainService`. This way a tenant could offboard itself partially or entirely from the platform without losing the provisioned infrastructure. This obviously depends on the ownership model of the infrastructure. +- `Run` clusters support scheduling workloads. A `Run` cluster may or may not also serve as `API` cluster. +- `API` clusters serve APIs but do not support scheduling workload (note that `API`/`Run` is a higher level platform concept) -## References +### References Projects with similar concepts: From 89021546e8209b577df8c7ab2bccd2ac875c1afe Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Wed, 5 Nov 2025 13:58:34 +0100 Subject: [PATCH 08/18] docs(serviceprovider): wip On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 196 +++++++++++-------- 1 file changed, 112 insertions(+), 84 deletions(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index 2a25484..3763c6b 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -4,7 +4,6 @@ - Define clear terminology around `ServiceProvider` in the OpenMCP space - Define `ServiceProvider` scope: responsibilities and boundaries of a `ServiceProvider` -- Define a `ServiceProvider` model that implements the higher level `API`/`Run` platform concept (to allow flexible deployment models, e.g. with `ClusterProvider` kcp) - Define `ServiceProvider` contract to implement `ServiceProvider` as a loosely coupled component in the openMCP context - Define how a `ServiceProvider` can be validated - (MCP) v1 learnings have been addressed @@ -12,63 +11,128 @@ ## Non-Goals - `ServiceProviders` does not need to deploy its `DomainService` on `WorkloadClusters`. For now a `DomainService` can be deployed on both `WorkloadCluster` or `MCPCluster`. +- Define a `ServiceProvider` model that implements a higher level `API`/`Run` platform concept (to allow flexible deployment models, e.g. with `ClusterProvider` kcp) -## Object Model +## Domain + +A `ServiceProvider` enables platform users to consume a `DomainServiceAPI` of a third party `DomainService`. A `DomainService` is a third party service that provides its service to end users in terms of a `DomainServiceAPI`. + +For example platform users want to use [Crossplane](https://www.crossplane.io/), specifically the `Object` API of [provider-kubernetes](https://github.com/crossplane-contrib/provider-kubernetes) to create Kubernetes objects on Kubernetes clusters they own without having to manage Crossplane themselves. So essentially consume `Crossplane` as a managed service of the `openMCP` platform. + +If we map this to the abstract terminology of a `DomainService` and `DomainServiceAPI`, this means users want to consume the `DomainServiceAPI` -> `Object` of a `DomainService` -> `Crossplane`. Note that `provider-kubernetes` depends on a running Crossplane installation to work properly. That is why `provider-kubernetes` itself can't be a `DomainService`. + +### Contracts + +#### End User Facing + +A `ServiceProviders` defines a `ServiceProviderAPI` to expose its managed service offering to end users. It is important to distinguish between `DomainServiceAPI` and `ServiceProviderAPI`. Both are end user facing but the `DomainServiceAPI` is the API that provides end user value while the `ServiceProviderAPI` defines the openMCP flavor of a `DomainService`. As with any other managed service offering, a `ServiceProvider` in openMCP restricts the usage of `DomainService` as part of his end user offering. A shifting range of platform supported `DomainService` versions that are available at a certain point in time is a typical example. + +```mermaid +graph LR + USR[User] + SPA[ServiceProviderAPI] + DSA[DomainServiceAPI] + USR --> SPA + USR --> DSA +``` + +#### Platform Operator Facing + +A `ServiceProvider` defines a `ServiceProviderConfig` to enable platform operators to define managed service offerings in terms of the available feature set of a `DomainService`. E.g. tenant 1 can consume `ServiceProviderAPI` `Crossplane` via a `ServiceProviderConfig` `A` that enables the installation of Crossplane versions `v1` and `v2`, while tenant 2 is only able to consume `Crossplane` version `v1` via `ServiceProviderConfig` `B`. + +```mermaid +graph LR + %% Operator + OP[Operator] + SP[ServiceProvider] + SPC[ServiceProviderConfig] + OP --> SP + OP --> SPC +``` + +:::info +Unlike a Crossplane `ProviderConfig` that is referenced by a `ManagedResource`, a `ServiceProviderConfig` is not directly referenced in `ServiceProviderAPI` objects. Consistent handling of `ServiceProviderConfig` across different `ServiceProviders` is still a goal we want to achieve to establish a consistent user and developer experience. +::: + +### Service Discovery + +End users need to be aware of a) the available service offerings and b) valid input values to consume a service offering. + +Both points are the responsibility of `PlatformServices` and not the `ServiceProvider` itself. + +A) is realized by installing the `ServiceProviderAPI` on the `OnboardingCluster`. + +B) is realized by communicating the `ServiceProviderConfig` options that are available to the user. + +One special kind of config information that a `ServiceProvider` needs are the artifact versions he can use to deploy a service. OpenMCP introduces the concept of `ReleaseChannels` that define the available artifacts (container images, helm charts, etc.) in context of an openMCP installation. A `ServiceProvider` indirectly consumes a `ReleaseChannel` via its `ServiceProviderConfig`: + +```mermaid +graph LR + RC[ReleaseChannel] + SPC[ServiceProviderConfig] + SP[ServiceProvider] + + %% edges + SP -->|one...uses...many|SPC + SPC -->|many...references subset of artifacts...one|RC +``` + +:::info +A `ServiceProviderConfig` may hold configuration parameters other than `ReleaseChannel` information/artifact versions. +::: + +### Deployment Model + +A `ServiceProvider` runs on the `PlatformCluster` and reconcile its `ServiceProviderAPI` on the `OnboardingCluster`. A `ServiceProvider` deploys a `DomainService` either on a `WorkloadCluster` or `MCPCluster` that reconciles the `DomainServiceAPI`. ```mermaid graph TD %% Onboarding Cluster - subgraph OnboardingCluster/API - SAPI[ServiceAPI] + subgraph OnboardingCluster + SPAPI[ServiceProviderAPI] end %% Platform Cluster - subgraph PlatformCluster/RUN + subgraph PlatformCluster SPO[openMCP-operator] SP[ServiceProvider] SPC[ServiceProviderConfig] end %% MCP Cluster - subgraph MCPCluster/RUN - DS[DomainService] + subgraph MCPCluster DSAPI[DomainServiceAPI] - SDSObject[SharedDomainServiceObject] - end - - %% WorkloadCluster - subgraph WorkloadCluster/RUN - SDS[SharedDomainService] end %% edges - SP -->|installs/reconciles|SAPI + SP -->|installs/reconciles|SPAPI SP -->|uses|SPC - SP -->|creates/updates/deletes|DS - SP -->|creates/updates/deletes|SDS - DS -->|installs/reconciles|DSAPI - SDS --->|creates/updates/deletes|SDSObject + SP -->|provides|DSAPI SPO-->|installs/reconciles|SP ``` -- The [openmcp-operator](https://github.com/openmcp-project/openmcp-operator) manages the lifecyclee of `ServiceProviders`. -- A `ServiceProvider` provides a third party service in tenant space. It manages the lifecycle of a `DomainService` and provides a `ServiceAPI`. -- The `ServiceProviderConfig` defines the platform facing config that is used in a reconcile run, e.g. to implement multi-tenancy on the `PlatformCluster`. -- The `ServiceAPI` is the tenant facing set of CRDs of a `ServiceProvider`. -- A `DomainService` is a third party service that manages `DomainServiceAPI` objects. -- The `DomainServiceAPI` is the set of CRDs introduced by `DomainService`. -- A `SharedDomainService` is a `DomainService` that can be deployed on a `WorkloadCluster`. A `SharedDomainService` does not expose its `DomainServiceAPI` to end users. -- A `SharedDomainServiceObject` is a Kubernets object managed by the `SharedDomainService`. +The `DomainServiceAPI` is either reconciled on the `MCPCluster` or a `WorkloadCluster`. The following diagram illustrates two simplified `DomainService` examples `Landscaper` and `Crossplane` with the corresponding `DomainServiceAPIs` `Installation` and `Bucket`. -## Domain +```mermaid +graph TD + %% Workload Cluster + subgraph WorkloadCluster + Landscaper + end -A `ServiceProvider` defines how a `DomainService` can be consumed by a tenant. It has the following responsibilities: + %% MCP Cluster + subgraph MCPCluster + Crossplane + Installation + Bucket + end -- Manage the lifecycle of a `DomainService` -- Define platform facing config -> `ServiceProviderConfig` -- Define a tenant facing `API` -> `ServiceConfig` + %% edges + Landscaper -->|reconciles|Installation + Crossplane -->|reconciles|Bucket +``` -A `DomainService` has to be treated as an external system where a `ServiceProvider` continuously has to prevent any drift caused by tenant actions on both `MCP` and `OnboardingCluster`. +- The [openmcp-operator](https://github.com/openmcp-project/openmcp-operator) manages the lifecyclee of `ServiceProviders`. ## Validation @@ -104,73 +168,37 @@ The following overview illustrates the layers of a `ServiceProvider` controller | Layer | Description | | :--- | :--- | -| Service Provider | defines `ServiceProviderAPI` and implements service-provider-runtime operations | +| Service Provider | defines `ServiceProviderAPI`/`ServiceProviderConfig` and implements service-provider-runtime operations | | service-provider-runtime | defines ServiceProvider reconciliation semantics | | multicluster/controller-runtime | defines generic reconciliation semantics | | Kubernetes API machinery | k8s essentials | -### Abstractions and Contracts +### Abstractions -Here we define the core interfaces that a consumer (`ServiceProvider` developer) has to implement, e.g. in Crossplane `ExternalConnector` creates `ExternalClient` which implements CRUD operations with `ExternalObservation`, `ExternalCreation`, etc. `Managed` interface defines what makes a k8s object a managed Crossplane resource, e.g. by referencing a `ProviderConfig`, specifying `ManagementPolicies`, `ConnectionSecrets`, etc. +In contrast to the API [contracts](#contracts) -## Out of Scope +Main tasks towards MCP/Workload Clusters based on watching the Onboarding Cluster APIs: -The remainder of this document contains topics that are out of scope for now. +- Create Service Deployment (Init Lifecycle) +- Observe Service Deployment (Drift Detection) +- Update Service Deployment (Reconcile Drift) +- Delete Service Deployment (End Lifecycle) -### Multicluster Execution Model - -Multi-cluster functionality for `ServiceProvider` is a design goal for future iterations and might get integrated into `service-provider-runtime`. This would enable service deployment on shared `WorkloadCluster`. The following two diagrams illustrate a `juggler` and `multicluster-runtime` version. - -#### Multicluster-runtime Facade - -```mermaid -graph TD - SPC[service-provider-controller] - SPR[service-provider-runtime] - MCR[multicluster-runtime] +Main tasks towards `PlatformCluster` based on `ServiceProviderConfig`: - %% edges - SPC -->|imports|SPR - SPR -->|imports/abstracts|MCR -``` - -Not feasable because this essentially means replacing/reimplementing existing domain controllers based on controller-runtime with multicluster-runtime. - -```mermaid -graph TD - %% WorkloadCluster - subgraph WorkloadCluster/RUN - DS[DomainService/RUN] - subgraph ServiceProviderInstance - SPR[service-provider-runtime] - end - end +- resolve and validate `ServiceProviderConfig` against e.g. available `ReleaseChannel` - %% MCPClusterA - subgraph MCPClusterA/API - DSAA[DomainServiceAPI] - end +Here we define the core interfaces that a consumer (`ServiceProvider` developer) has to implement, e.g. in Crossplane `ExternalConnector` creates `ExternalClient` which implements CRUD operations with `ExternalObservation`, `ExternalCreation`, etc. `Managed` interface defines what makes a k8s object a managed Crossplane resource, e.g. by referencing a `ProviderConfig`, specifying `ManagementPolicies`, `ConnectionSecrets`, etc. - %% MCPClusterB - subgraph MCPClusterB/API - DSAB[DomainServiceAPI] - end +## Out of Scope - %% MCPClusterC - subgraph MCPClusterC/API - DSAC[DomainServiceAPI] - end +The remainder of this document contains topics that are out of scope for now. - %% edges - DS ---|forwardRequests/syncStatus| SPR - SPR -->|reconciles|DSAA - SPR -->|reconciles|DSAB - SPR -->|reconciles|DSAC -``` +### Multicluster Execution Model -#### Object Syncer / Juggler +Multi-cluster functionality for `ServiceProvider` is a design goal for future iterations and might get integrated into `service-provider-runtime`. This would enable service deployment on shared `WorkloadCluster`. -Another approach would be to sync API objects between `API` and `RUN` clusters as a feature of service-provider-runtime. +An approach could be to sync API objects between `API` and `RUN` clusters as a feature of service-provider-runtime. ```mermaid graph TD From 9d9f0b4552fe724fc56787dad99caeb78caf4f69 Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Thu, 6 Nov 2025 18:45:16 +0100 Subject: [PATCH 09/18] docs(serviceprovider): typo On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index 3763c6b..38f9a3a 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -132,7 +132,7 @@ graph TD Crossplane -->|reconciles|Bucket ``` -- The [openmcp-operator](https://github.com/openmcp-project/openmcp-operator) manages the lifecyclee of `ServiceProviders`. +- The [openmcp-operator](https://github.com/openmcp-project/openmcp-operator) manages the lifecycle of `ServiceProviders`. ## Validation From 41b134c435708a3568a0eaaaa39248bfcb96f9ee Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Thu, 6 Nov 2025 18:49:31 +0100 Subject: [PATCH 10/18] docs(serviceprovider): wip On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index 38f9a3a..abe3239 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -177,7 +177,7 @@ The following overview illustrates the layers of a `ServiceProvider` controller In contrast to the API [contracts](#contracts) -Main tasks towards MCP/Workload Clusters based on watching the Onboarding Cluster APIs: +Main tasks towards MCP/Workload Clusters based on watching the `ServiceProviderAPI`: - Create Service Deployment (Init Lifecycle) - Observe Service Deployment (Drift Detection) From 55e0f03a91b5b70c099010d891d62549a33d4b06 Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Thu, 6 Nov 2025 18:59:55 +0100 Subject: [PATCH 11/18] docs(serviceprovider): wip On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index abe3239..df30fe4 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -25,7 +25,7 @@ If we map this to the abstract terminology of a `DomainService` and `DomainServi #### End User Facing -A `ServiceProviders` defines a `ServiceProviderAPI` to expose its managed service offering to end users. It is important to distinguish between `DomainServiceAPI` and `ServiceProviderAPI`. Both are end user facing but the `DomainServiceAPI` is the API that provides end user value while the `ServiceProviderAPI` defines the openMCP flavor of a `DomainService`. As with any other managed service offering, a `ServiceProvider` in openMCP restricts the usage of `DomainService` as part of his end user offering. A shifting range of platform supported `DomainService` versions that are available at a certain point in time is a typical example. +A `ServiceProviders` defines a `ServiceProviderAPI` to expose its managed service offering to end users. It is important to distinguish between `DomainServiceAPI` and `ServiceProviderAPI`. Both are end user facing but the `DomainServiceAPI` is the API that provides end user value while the `ServiceProviderAPI` defines the openMCP flavor of a `DomainService`. As with any other managed service offerings, a `ServiceProvider` in openMCP restricts the usage of `DomainService` to a subset of features. A simple example is a shifting range of platform supported `DomainService` versions that are available at a certain point. ```mermaid graph LR From 2619c5376abf244671c956e531ee114e529ac809 Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Fri, 7 Nov 2025 10:07:20 +0100 Subject: [PATCH 12/18] docs(serviceprovider): wip On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index df30fe4..f287cfb 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -60,25 +60,31 @@ End users need to be aware of a) the available service offerings and b) valid in Both points are the responsibility of `PlatformServices` and not the `ServiceProvider` itself. -A) is realized by installing the `ServiceProviderAPI` on the `OnboardingCluster`. +A) is realized by installing the `ServiceProviderAPI` on the `OnboardingCluster`. Any platform tenant is aware of any available `ServiceProviderAPI`. In other words the platform doesn't hide its end user facing feature set depending on whether a user belongs to a tenant that is able to successfully consume a `ServiceProviderAPI`. -B) is realized by communicating the `ServiceProviderConfig` options that are available to the user. +B) is realized by communicating the `ServiceProviderConfig` options that are available to the user. A user of a tenant without a `ServiceProviderConfig` can consume a `ServiceProviderAPI` but `DomainService` deployment will be denied. One special kind of config information that a `ServiceProvider` needs are the artifact versions he can use to deploy a service. OpenMCP introduces the concept of `ReleaseChannels` that define the available artifacts (container images, helm charts, etc.) in context of an openMCP installation. A `ServiceProvider` indirectly consumes a `ReleaseChannel` via its `ServiceProviderConfig`: ```mermaid -graph LR - RC[ReleaseChannel] - SPC[ServiceProviderConfig] - SP[ServiceProvider] +graph TD + subgraph Platform + subgraph ServiceProvider + SPC[ServiceProviderConfig] + SP[ServiceProvider] + end + TN[Tenant] + RC[ReleaseChannel] + end %% edges SP -->|one...uses...many|SPC SPC -->|many...references subset of artifacts...one|RC + SPC ---|many...allows access...many|TN ``` :::info -A `ServiceProviderConfig` may hold configuration parameters other than `ReleaseChannel` information/artifact versions. +A `ServiceProviderConfig` may hold configuration parameters other than `ReleaseChannel` information/artifact versions. In that sense it is more than just a 'version pinning' mechanism. ::: ### Deployment Model From 0e9b9dc681b0b360e813cc53548e881009501e6d Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Fri, 7 Nov 2025 10:43:42 +0100 Subject: [PATCH 13/18] docs(serviceprovider): wip On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index f287cfb..08cc768 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -161,14 +161,7 @@ tbd. A runtime is a collection of abstractions and contracts that provides an environment in which user-defined logic is executed. -The service provider runtime is built on top of controller-runtime and provides a service provider specific reconciliation loop. - -It provides: - -- client abstractions (in xp external clients, in openmcp e.g. reuse common juggler reconcilers like flux?) -- lifecycle management abstractions of `ServiceProviderAPI` objects (the reconcile loop) -- platform specific features (in xp e.g. late initialize, external-name and pause annotations), enables us to implement platform features for all service providers (a `ServiceProvider` only needs to update their runtime dependency) -- handling of cross-cutting concerns like event recording, logging, metrics, rate limits +The service provider runtime is built on top of controller-runtime and provides a service provider specific reconciliation loop. It gives us as a platform the possibility to implement platform specific feature around service providers that is are not `DomainService` specific. This way we provide a consistent experience to both end users and developers when working with `ServiceProviders`. The following overview illustrates the layers of a `ServiceProvider` controller a simplified way: @@ -248,11 +241,8 @@ graph TD ### Ideas -- `SoftDelete` platform concept. A `managed` service can transition to a `unmanaged` service by soft deleting its corresponding `ServiceProviderAPI` or the `ServiceProvider` entirely without losing the `DomainService`. This way a tenant could offboard itself partially or entirely from the platform without losing the provisioned infrastructure. This obviously depends on the ownership model of the infrastructure. -- In the above model the `OnboardingCluster` is a continuous `API` cluster. We might want to provision dedicated or shared tenant `API` servers (e.g. with `ClusterProvider` kcp) based on some kind of component discovery that lets the tenant pick its feature/component set. This way the `OnboardingCluster` is only used to onboard new tenants. And we don't run into CRD management hell/bottlenecks. -- Another thought regarding the `OnboardingCluster`. If we introduce tenant `API` clusters, they could be used to create MCPs. This again implies that instead of having the `OnboardingCluster` create `MCPs`, we might want to have the `OnboardingCluster` create `Tenants` as the entry point for users -> start with an identity object like `Tenant` or `Account` instead of a usage artifact like `MCP`. +- `SoftDelete` platform concept. A `managed` service can transition to a `unmanaged` service by soft deleting its corresponding `ServiceProviderConfig` without losing the `DomainService`. This way a tenant could offboard itself partially or entirely from the platform without losing the provisioned infrastructure. This obviously depends on the ownership model of the infrastructure. - Distinguish between `Run` and `API` artifacts on all platform layers -- Illustrate different deployment models with `Run`/`API` concept ### Terminology From b567eeb16b0009f5dfb215cc8b8ece14c7ade02c Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Fri, 7 Nov 2025 10:45:55 +0100 Subject: [PATCH 14/18] docs(serviceprovider): wip On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index 08cc768..0dc0436 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -161,7 +161,7 @@ tbd. A runtime is a collection of abstractions and contracts that provides an environment in which user-defined logic is executed. -The service provider runtime is built on top of controller-runtime and provides a service provider specific reconciliation loop. It gives us as a platform the possibility to implement platform specific feature around service providers that is are not `DomainService` specific. This way we provide a consistent experience to both end users and developers when working with `ServiceProviders`. +The `service-provider-runtime` is built on top of `controller-runtime` and provides a service provider specific reconciliation loop. It gives us as a platform the possibility to implement platform specific features around service providers that are not `DomainService` specific. This way we provide a consistent experience to both end users and developers when working with `ServiceProviders`. The following overview illustrates the layers of a `ServiceProvider` controller a simplified way: From 391e840830de6da901ca9d38c7ee25eaf59fd0ba Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Fri, 7 Nov 2025 19:26:06 +0100 Subject: [PATCH 15/18] docs(serviceprovider): wip On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 46 ++++++++++++++------ 1 file changed, 33 insertions(+), 13 deletions(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index 0dc0436..3aec043 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -159,18 +159,18 @@ tbd. ## Runtime -A runtime is a collection of abstractions and contracts that provides an environment in which user-defined logic is executed. +A runtime is a collection of abstractions and contracts that provides an environment in which user-defined logic is executed. This introduces a clear separation between `ServiceProvider` developer domain and platform developer domain. -The `service-provider-runtime` is built on top of `controller-runtime` and provides a service provider specific reconciliation loop. It gives us as a platform the possibility to implement platform specific features around service providers that are not `DomainService` specific. This way we provide a consistent experience to both end users and developers when working with `ServiceProviders`. +The `service-provider-runtime` is built on top of `controller-runtime` and provides a service provider specific reconciliation loop. It gives us as a platform the possibility to implement platform specific features around service providers. At the same time `ServiceProvider` developers can focus on `DomainService` specific logic without the burden to understand platform internals. This way we provide a consistent experience to both end users and developers when working with `ServiceProviders`. The following overview illustrates the layers of a `ServiceProvider` controller a simplified way: -| Layer | Description | -| :--- | :--- | -| Service Provider | defines `ServiceProviderAPI`/`ServiceProviderConfig` and implements service-provider-runtime operations | -| service-provider-runtime | defines ServiceProvider reconciliation semantics | -| multicluster/controller-runtime | defines generic reconciliation semantics | -| Kubernetes API machinery | k8s essentials | +| Layer | Description | Target Audience | +| :--- | :--- | :--- | +| Service Provider | defines `ServiceProviderAPI`/`ServiceProviderConfig` and implements service-provider-runtime operations | service provider developers | +| service-provider-runtime | defines ServiceProvider reconciliation semantics | platform developers | +| multicluster/controller-runtime | defines generic reconciliation semantics | out of scope | +| Kubernetes API machinery | k8s essentials | out of scope | ### Abstractions @@ -178,16 +178,36 @@ In contrast to the API [contracts](#contracts) Main tasks towards MCP/Workload Clusters based on watching the `ServiceProviderAPI`: -- Create Service Deployment (Init Lifecycle) -- Observe Service Deployment (Drift Detection) -- Update Service Deployment (Reconcile Drift) -- Delete Service Deployment (End Lifecycle) +- Observe Service Deployment (Drift Detection) -> IN: context, apiObject, reconcileScope; OUT: bool[exists, drift], error +- Create Service Deployment (Init Lifecycle) -> IN: context, apiObject, reconcileScope; OUT: error +- Update Service Deployment (Reconcile Drift) -> IN: context, apiObject, reconcileScope; OUT: error +- Delete Service Deployment (End Lifecycle) -> IN: context, apiObject, reconcileScope; OUT: error + +where `reconcileScope` holds the `ServiceProviderConfig` and clients to access onboarding, mcp and workload clusters. Main tasks towards `PlatformCluster` based on `ServiceProviderConfig`: - resolve and validate `ServiceProviderConfig` against e.g. available `ReleaseChannel` -Here we define the core interfaces that a consumer (`ServiceProvider` developer) has to implement, e.g. in Crossplane `ExternalConnector` creates `ExternalClient` which implements CRUD operations with `ExternalObservation`, `ExternalCreation`, etc. `Managed` interface defines what makes a k8s object a managed Crossplane resource, e.g. by referencing a `ProviderConfig`, specifying `ManagementPolicies`, `ConnectionSecrets`, etc. +### Reconcile Sequence + +```mermaid +sequenceDiagram + %% participants + participant cr as controller-runtime + participant spr as service-provider-runtime + participant sp as service-provider + + %% messages + cr->>spr: reconcile + spr->>spr: set up reconcileScope + spr-->>cr: end if no service provider config exists + spr->>spr: fetch API object from onboarding cluster + spr->>sp: observe(apiObject, reconcileScope) + sp-->>spr: exists/drift + spr->>sp: create/update/delete(apiObject, reconcileScope) + spr-->>cr: requeueAfter +``` ## Out of Scope From fe56d51b054fe3b26d18f9a748b65fd77c4a17fe Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Fri, 7 Nov 2025 19:44:10 +0100 Subject: [PATCH 16/18] docs(serviceprovider): wip On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index 3aec043..2ea4325 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -209,6 +209,10 @@ sequenceDiagram spr-->>cr: requeueAfter ``` +:::info +This sequence expects that a `ServiceProviderConfig` does not require validation by a `ServiceProvider`. How valid `ServiceProviderConfigs` are generated in the `PlatformCluster` is tbd. +::: + ## Out of Scope The remainder of this document contains topics that are out of scope for now. From b0881672c8711d66d8ddae5f1ec8d8405454f69e Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Fri, 7 Nov 2025 19:53:37 +0100 Subject: [PATCH 17/18] docs(serviceprovider): wip On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index 2ea4325..c0886c8 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -219,7 +219,7 @@ The remainder of this document contains topics that are out of scope for now. ### Multicluster Execution Model -Multi-cluster functionality for `ServiceProvider` is a design goal for future iterations and might get integrated into `service-provider-runtime`. This would enable service deployment on shared `WorkloadCluster`. +Multi-cluster functionality for `ServiceProvider` is a design goal for future iterations and might get integrated into `service-provider-runtime`. This would generally enable to run any `DomainService` on shared `WorkloadCluster`. An approach could be to sync API objects between `API` and `RUN` clusters as a feature of service-provider-runtime. From a07f374ad632dad60b064d189ccba7627ba0ef23 Mon Sep 17 00:00:00 2001 From: Christopher Junk Date: Fri, 7 Nov 2025 19:58:34 +0100 Subject: [PATCH 18/18] docs(serviceprovider): wip On-behalf-of: @SAP christopher.junk@sap.com Signed-off-by: Christopher Junk --- docs/about/design/service-provider-design.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/about/design/service-provider-design.md b/docs/about/design/service-provider-design.md index c0886c8..52ce771 100644 --- a/docs/about/design/service-provider-design.md +++ b/docs/about/design/service-provider-design.md @@ -19,13 +19,13 @@ A `ServiceProvider` enables platform users to consume a `DomainServiceAPI` of a For example platform users want to use [Crossplane](https://www.crossplane.io/), specifically the `Object` API of [provider-kubernetes](https://github.com/crossplane-contrib/provider-kubernetes) to create Kubernetes objects on Kubernetes clusters they own without having to manage Crossplane themselves. So essentially consume `Crossplane` as a managed service of the `openMCP` platform. -If we map this to the abstract terminology of a `DomainService` and `DomainServiceAPI`, this means users want to consume the `DomainServiceAPI` -> `Object` of a `DomainService` -> `Crossplane`. Note that `provider-kubernetes` depends on a running Crossplane installation to work properly. That is why `provider-kubernetes` itself can't be a `DomainService`. +If we map this to the terminology of a `DomainService` and `DomainServiceAPI`, this means users want to consume the `DomainServiceAPI` -> `Object` of a `DomainService` -> `Crossplane`. Note that `provider-kubernetes` depends on a running Crossplane installation to work properly. That is why `provider-kubernetes` itself can't be a `DomainService`. ### Contracts #### End User Facing -A `ServiceProviders` defines a `ServiceProviderAPI` to expose its managed service offering to end users. It is important to distinguish between `DomainServiceAPI` and `ServiceProviderAPI`. Both are end user facing but the `DomainServiceAPI` is the API that provides end user value while the `ServiceProviderAPI` defines the openMCP flavor of a `DomainService`. As with any other managed service offerings, a `ServiceProvider` in openMCP restricts the usage of `DomainService` to a subset of features. A simple example is a shifting range of platform supported `DomainService` versions that are available at a certain point. +A `ServiceProvider` defines a `ServiceProviderAPI` to expose its managed service offering to end users. It is important to distinguish between `DomainServiceAPI` and `ServiceProviderAPI`. Both are end user facing but the `DomainServiceAPI` is the API that provides end user value while the `ServiceProviderAPI` defines the openMCP flavor of a `DomainService`. As with any other managed service offerings, a `ServiceProvider` in openMCP restricts the usage of `DomainService` to a subset of features. A simple example is a shifting range of platform supported `DomainService` versions that are available at a certain point. ```mermaid graph LR