Tracking issue for chained upgrades

[fabriziopandini]

Tracking issue for #12199

• ✨ Implement core logic for chained upgrades #12726 (including fields in CC and refactoring the E2E tests)
  • Follow-up: ⚠️ Improve chained upgrade observability #12973
    • conditions in general: audit relevant parts in topology/cluster/conditions.go, we should also especially surface it in conditions if the newly introduced hooks are blocking
    • audit logs for correctness in desired_state.go (e.g. the versions we log)
    • ensure we send the right versions in hooks
  • Follow-up: Add additional validation in Cluster/ClusterClass webhook: [help wanted] (@Karthik-K-N)
    • Cluster webhook: ClusterClass rebase: check that new ClusterClass includes .spec.topology.version (if CC has versions)
    • ClusterClass webhook: on create/update: ensure versions of all Clusters using that ClusterClass are included in ClusterClass versions (if CC has versions)
  • Follow-up: 🌱 Allow >1 minor version upgrades if generateUpgradePlan extension is defined #12979
• Integration with Runtime Extension [help wanted] (@sivchari)
  • Add .spec.upgrade.external.generateUpgradePlanExtension field to ClusterClass: ✨ Add .spec.upgrade.external.generateUpgradePlanExtension field to ClusterClass #12809
  • Add GenerateUpgradePlanRequest/GenerateUpgradePlanResponse types + GenerateUpgradePlan hook ✨ Add types and hook for GenerateUpgradePlan #12823
  • Call GenerateUpgradePlan Runtime Extension in desired_state.go: ✨ Call GenerateUpgradePlanRequest Runtime Extension #12903
  • Implement GenerateUpgradePlan in test/extension: ✨ Implement GenerateUpgradePlan handler #12927
  • Add corresponding e2e test coverage: ✨ Change RuntimeSDK e2e test ClusterClass to use GenerateUpgradePlan extension #12955
  • Double check instructions in comment below
• New lifecycle hooks (@fabriziopandini)
  • Add hooks types, change existing hooks where required ✨ Add lifecycle hooks for chained-upgrade #12878
  • Implement hooks calls: ✨ Call new lifecycle hooks for chained-upgrades #12891
    • Extend e2e test, including TODO(chained-upgrade): FP, we can't check next CP upgrade doesn't start, it starts immediately (ideally we have e2e test coverage for all upgrade hooks)
  • Make the AfterClusterUpgrade hook blocking for the next upgrade: ⚠️ Make the AfterClusterUpgrade hook blocking #12984
• "Testing"
  • Brainstorm about edge cases
    • Think through all permutations of setting Kubernetes versions / external.upgrade on CC (and which ones we cover in e2e tests)
  • Test chained upgrade without workers: ✨ Call new lifecycle hooks for chained-upgrades #12891 (comment)
  • Go through a few upgrade flows (desired state + upgrade plan) (either just by looking through the code or with the debugger)
• Misc
  • Audit TODO(chained-upgrade)
  • Ensure strict sequencing for all hooks except AfterControlPlaneInitialized (i.e. we have to wait for hook completion before continuing), Also take a look at reentrancy and see if we can improve it with a reasonable complexity trade-off (e.g. mark as pending: AfterControlPlaneUpgrade)
• Documentation
  • Update lifecycle hook documentation in the book
  • Sync & check against the proposal that everything was either implemented or is tracked in a follow-up issue

Next release:

• Consider if to add more info about version(s) in status
• Consider performance optimizations for topology controller (specifically also for lifecycle hook calls), e.g. rate-limiting
• Introduce util to deduplicate Machine version checks in RuntimeSDK e2e test: ✨ Call new lifecycle hooks for chained-upgrades #12891 (comment)
• Follow-up: Update desired_state_test.go for new replica fields (e.g. remove unavailableReplicas)
• Follow-up: Improve GetUpgradePlanFromClusterClassVersions algo: ✨ Implement core logic for chained upgrades #12726 (comment)
• Follow-up: Reduce dependency tree for util patch ⚠️ Make the AfterClusterUpgrade hook blocking #12984 (comment)

[sbueringer]

@Karthik-K-N @sivchari Would you be interested in taking over one of these tasks?

• Add additional validation in Cluster/ClusterClass webhook (small'ish)
• Integration with Runtime Extension (a bit bigger)

If you're interested we will provide more details

[Karthik-K-N]

Yep, Sure, will take that up.

[sivchari]

Thanks, I'm willing to do it. Thinking to take over Integration with Runtime Extension

[fabriziopandini]

@sivchari About Integration with Runtime Extension [help wanted]

Add .spec.upgrade.external.generateUpgradePlanExtension field to ClusterClass (+corresponding validation)

  spec:
  upgrade:
    external:    
      generateUpgradePlanExtension: get-upgrade-plan.foo

MinLength=1, MaxLength=512, +optional

• Add GenerateUpgradePlanRequest/GenerateUpgradePlanResponse types + GenerateUpgradePlan hook

under api/runtime/hooks/v1alpha1 add a new upgrade_plan_types.go , prior art GeneratePatches in topologymutation_types.go

Messages will be

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: GenerateUpgradePlanRequest
settings: <Runtime Extension settings>
cluster:
  apiVersion: cluster.x-k8s.io/v1beta1
  kind: Cluster
  metadata:
    name: test-cluster
    namespace: test-ns
  spec:
    ...
  status:
    ...
fromControlPlaneKubernetesVersion: "v1.29.0"
fromWorkersKubernetesVersion: "v1.29.0"
toKubernetesVersion: "v1.33.0"

• Settings comes from embedding CommonRequest
• Cluster is clusterv1beta1.Cluster

And

apiVersion: hooks.runtime.cluster.x-k8s.io/v1alpha1
kind: GenerateUpgradePlanResponse
status: Success # or Failure
message: "error message if status == Failure"
controlPlaneUpgrades:
- version: v1.30.0
- version: v1.31.0
- version: v1.32.3
workersUpgrades:
- version: v1.30.0
- version: v1.32.3

• status and message comes from embedding CommonResponse

  • Call GenerateUpgradePlanRequest Runtime Extension in desired_state.go
    The call must be implemented after

    cluster-api/exp/topology/desiredstate/desired_state.go

    Lines 123 to 125 in 9db4c78

    	if len(s.Blueprint.ClusterClass.Spec.KubernetesVersions) > 0 {
    	getUpgradePlan = GetUpgradePlanFromClusterClassVersions(s.Blueprint.ClusterClass.Spec.KubernetesVersions)
    	}

    , only if the extension is defined in CC (the runtime extension takes precedence if defined).

The implementation of this must be done in a separate func, to be added to upgrade_plan.go after GetUpgradePlanOneMinor and GetUpgradePlanFromClusterClassVersions, e.g

func GetUpgradePlanFromExtension(runtimeClient runtimeclient.Client) func(_ context.Context, desiredVersion, currentControlPlaneVersion, _ string) ([]string, []string, error) {
	return func(_ context.Context, desiredVersion, currentControlPlaneVersion, _ string) ([]string, []string, error) {
		...
	}
}

the body of this func must call the new runtime extension, similar to

cluster-api/internal/controllers/topology/cluster/patches/external/external_patch_generator.go

Lines 47 to 67 in ed4e6b2

	func (e externalPatchGenerator) Generate(ctx context.Context, forObject client.Object, req *runtimehooksv1.GeneratePatchesRequest) (*runtimehooksv1.GeneratePatchesResponse, error) {
	if !feature.Gates.Enabled(feature.RuntimeSDK) {
	return nil, errors.Errorf("can not use external patch %q if RuntimeSDK feature flag is disabled", e.patch.External.GeneratePatchesExtension)
	}
	// Set the settings defined in external patch definition on the request object.
	// These settings will override overlapping keys defined in ExtensionConfig settings.
	req.Settings = e.patch.External.Settings
	// The req object is re-used across multiple calls to the patch generator.
	// Reset the settings so that the request does not remain modified here.
	defer func() {
	req.Settings = nil
	}()
	resp := &runtimehooksv1.GeneratePatchesResponse{}
	err := e.runtimeClient.CallExtension(ctx, runtimehooksv1.GeneratePatches, forObject, e.patch.External.GeneratePatchesExtension, req, resp)
	if err != nil {
	return nil, err
	}
	return resp, nil
	}

• Implement GenerateUpgradePlan in test/extension

create a new upgradeplan folder under test/extension/handlers

The implemenatation should be a simplified version of test/extension/handlers/lifecycle/handlers.go; the answer to the request must be created by calling GetUpgradePlanFromClusterClassVersions with in inout the list of versions that you get from GetKubernetesVersions in the kind mapper.

• Add corresponding e2e test coverage (another ClusterUpgradeWithRuntimeSDKSpec test)

The test must be a copy of "When performing chained upgrades for workload cluster using ClusterClass in a different NS with RuntimeSDK [ClusterClass]" but with a new flavor, that diffrers from "upgrades-runtimesdk" only because the Cluster class has the new generateUpgradePlanExtension field set