Merge pull request #99890 from bergerhoffer/kueue-migration-1.1-integration

Kueue migration 1.1 integration

Author: bergerhoffer

_attributes/common-attributes.adoc

@@ -79,7 +79,10 @@ endif::[]
 :descheduler-operator: Kube Descheduler Operator
 :cli-manager: CLI Manager Operator
 :lws-operator: Leader Worker Set Operator
-:kueue-prod-name: Red{nbsp}Hat build of Kueue
+//Kueue
+:kueue-name: Red{nbsp}Hat build of Kueue
+:kueue-op: Red Hat Build of Kueue Operator
+:ms: Red{nbsp}Hat build of MicroShift (MicroShift)
 // Backup and restore
 :launch: image:app-launcher.png[title="Application Launcher"]
 :mtc-first: Migration Toolkit for Containers (MTC)

_topic_maps/_topic_map.yml

@@ -3432,6 +3432,34 @@ Distros: openshift-enterprise
 Topics:
 - Name: Overview of AI workloads on OpenShift Container Platform
   File: index
+- Name: Red Hat build of Kueue
+  Dir: kueue
+  Distros: openshift-enterprise
+  Topics:
+  - Name: Introduction to Red Hat build of Kueue
+    File: about-kueue
+  - Name: Release notes
+    File: release-notes
+  - Name: Installing Red Hat build of Kueue
+    File: install-kueue
+  - Name: Installing Red Hat build of Kueue in a disconnected environment
+    File: install-disconnected
+  - Name: Configuring role-based permissions
+    File: rbac-permissions
+  - Name: Configuring quotas
+    File: configuring-quotas
+  - Name: Managing jobs and workloads
+    File: managing-workloads
+  - Name: Using cohorts
+    File: using-cohorts
+  - Name: Configuring fair sharing
+    File: configuring-fairsharing
+  - Name: Gang scheduling
+    File: gangscheduling
+  - Name: Running jobs with quota limits
+    File: running-kueue-jobs
+  - Name: Getting support
+    File: getting-support
 - Name: Leader Worker Set Operator
   Dir: leader_worker_set
   Distros: openshift-enterprise

ai_workloads/index.adoc

@@ -15,6 +15,7 @@ include::modules/ai-operators.adoc[leveloffset=+1]
 [role="_additional-resources"]
 .Additional resources
 
+* xref:../ai_workloads/kueue/about-kueue.adoc#about-kueue[Introduction to {kueue-name}]
 * xref:../ai_workloads/leader_worker_set/index.adoc#lws-about[{lws-operator} overview]
 
 // Exclude this for now until we can get it reviewed by the RHOAI team

ai_workloads/kueue/_attributes

@@ -0,0 +1 @@
+../../_attributes/

ai_workloads/kueue/about-kueue.adoc

@@ -0,0 +1,52 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="about-kueue"]
+= Introduction to {kueue-name}
+:context: about-kueue
+
+toc::[]
+
+{kueue-name} is a Kubernetes-native system that manages access to resources for jobs.
+{kueue-name} can determine when a job waits, is admitted to start by creating pods, or should be _preempted_, meaning that active pods for that job are deleted.
+
+[NOTE]
+====
+In the context of {kueue-name}, a job can be defined as a one-time or on-demand task that runs to completion.
+====
+
+{kueue-name} is based on the link:https://kueue.sigs.k8s.io/docs/[Kueue] open source project.
+
+{kueue-name} is compatible with environments that use heterogeneous, elastic resources. This means that the environment has many different resource types, and those resources are capable of dynamic scaling.
+
+{kueue-name} does not replace any existing components in a Kubernetes cluster, but instead integrates with the existing Kubernetes API server, scheduler, and cluster autoscaler components.
+
+{kueue-name} supports all-or-nothing semantics. This means that either an entire job with all of its components is admitted to the cluster, or the entire job is rejected if it does not fit on the cluster.
+
+// Personas
+[id="about-kueue-personas"]
+== Personas
+
+Different personas exist in a {kueue-name} workflow.
+
+Batch administrators:: Batch administrators manage the cluster infrastructure and establish quotas and queues.
+Batch users:: Batch users run jobs on the cluster. Examples of batch users might be researchers, AI/ML engineers, or data scientists.
+Serving users:: Serving users run jobs on the cluster. For example, to expose a trained AI/ML model for inference.
+Platform developers:: Platform developers integrate {kueue-name} with other software. They might also contribute to the Kueue open source project.
+
+[id="about-kueue-workflow"]
+== Workflow overview
+
+The {kueue-name} workflow can be described at a high level as follows:
+
+. Batch administrators create and configure `ResourceFlavor`, `LocalQueue`, and `ClusterQueue` resources.
+. User personas create jobs on the cluster.
+. The Kubernetes API server validates and accepts job data.
+. {kueue-name} admits jobs based on configured options, such as order or quota. It injects affinity into the job by using resource flavors, and creates a `Workload` object that corresponds to each job.
+. The applicable controller for the job type creates pods.
+. The Kubernetes scheduler assigns pods to a node in the cluster.
+. The Kubernetes cluster autoscaler provisions more nodes as required.
+
+////
+TODO:Add docs explaining different job / workload types
+These can be added as we add stories / docs for different use cases
+////

ai_workloads/kueue/configuring-fairsharing.adoc

@@ -0,0 +1,18 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="configuring-fairsharing"]
+= Configuring fair sharing
+:context: configuring-fairsharing
+
+toc::[]
+
+Fair sharing is a preemption strategy that is used to achieve an equal or weighted share of borrowable resources between the tenants of a cohort. Borrowable resources are the unused nominal quota of all the cluster queues in a cohort.
+
+You can configure fair sharing by setting the `preemptionPolicy` value in the `Kueue` custom resource (CR) to `FairSharing`.
+
+include::modules/kueue-clusterqueue-share-value.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+[id="additional-resources_{context}"]
+== Additional resources
+* xref:../../ai_workloads/kueue/install-kueue.adoc#create-kueue-cr_install-kueue[Creating a `Kueue` custom resource]

ai_workloads/kueue/configuring-quotas.adoc

@@ -0,0 +1,38 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="configuring-quotas"]
+= Configuring quotas
+:context: configuring-quotas
+
+toc::[]
+
+As an administrator, you can use {kueue-name} to configure quotas to optimize resource allocation and system throughput for user workloads.
+You can configure quotas for compute resources such as CPU, memory, pods, and GPU.
+
+You can configure quotas in {kueue-name} by completing the following steps:
+
+. Configure a cluster queue.
+. Configure a resource flavor.
+. Configure a local queue.
+
+Users can then submit their workloads to the local queue.
+
+include::modules/kueue-configuring-clusterqueues.adoc[leveloffset=+1]
+
+[role="_next-steps"]
+[id="clusterqueues-next-steps_{context}"]
+.Next steps
+
+The cluster queue is not ready for use until a xref:../../ai_workloads/kueue/configuring-quotas.adoc#configuring-resourceflavors_configuring-quotas[`ResourceFlavor` object] has also been configured.
+
+include::modules/kueue-configuring-resourceflavors.adoc[leveloffset=+1]
+
+include::modules/kueue-configuring-localqueues.adoc[leveloffset=+1]
+
+include::modules/kueue-configuring-localqueue-defaults.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+[id="clusterqueues-additional-resources_{context}"]
+== Additional resources
+* xref:../../ai_workloads/kueue/rbac-permissions.adoc#rbac-permissions[RBAC permissions]
+* link:https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/[Kubernetes documentation about cluster queues]

ai_workloads/kueue/gangscheduling.adoc

@@ -0,0 +1,27 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="gangscheduling"]
+= Gang scheduling
+:context: gangscheduling
+
+toc::[]
+
+Gang scheduling ensures that a group or _gang_ of related jobs only start when all required resources are available. {kueue-name} enables gang scheduling by suspending jobs until the {product-title} cluster can guarantee the capacity to start and execute all of the related jobs in the gang together. This is also known as _all-or-nothing_ scheduling.
+
+Gang scheduling is important if you are working with expensive, limited resources, such as GPUs. Gang scheduling can prevent jobs from claiming but not using GPUs, which can improve GPU utilization and can reduce running costs. Gang scheduling can also help to prevent issues like resource segmentation and deadlocking.
+
+include::modules/kueue-configuring-gangscheduling.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+[id="additional-resources_{context}"]
+== Additional resources
+* xref:../../ai_workloads/kueue/install-kueue.adoc#create-kueue-cr_install-kueue[Creating a Kueue custom resource]
+
+////
+// use case - deep learning
+One classic example is in deep learning workloads. Deep learning frameworks (Tensorflow, PyTorch etc) require all the workers to be running during the training process.
+
+In this scenario, when you deploy training workloads, all the components should be scheduled and deployed to ensure the training works as expected.
+
+Gang Scheduling is a critical feature for Deep Learning workloads to enable all-or-nothing scheduling capability, as most DL frameworks requires all workers to be running to start training process. Gang Scheduling avoids resource inefficiency and scheduling deadlock sometimes.
+////

ai_workloads/kueue/getting-support.adoc

@@ -0,0 +1,27 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="getting-support"]
+= Getting support
+:context: getting-support
+
+toc::[]
+
+If you experience difficulty with a procedure described in this documentation, or with {kueue-name} in general, visit the link:http://access.redhat.com[Red{nbsp}Hat Customer Portal].
+
+From the Customer Portal, you can:
+
+* Search or browse through the Red{nbsp}Hat Knowledgebase of articles and solutions relating to Red{nbsp}Hat products.
+* Submit a support case to Red{nbsp}Hat Support.
+* Access other product documentation.
+
+[id="getting-support-rh-kb"]
+== About the Red Hat Knowledgebase
+
+The link:https://access.redhat.com/knowledgebase[Red{nbsp}Hat Knowledgebase] provides rich content aimed at helping you make the most of Red{nbsp}Hat's products and technologies. The Red{nbsp}Hat Knowledgebase consists of articles, product documentation, and videos outlining best practices on installing, configuring, and using Red{nbsp}Hat products. In addition, you can search for solutions to known issues, each providing concise root cause descriptions and remedial steps.
+
+include::modules/kueue-gathering-cluster-data.adoc[leveloffset=+1]
+
+[id="getting-support-additional-resources"]
+[role="_additional-resources"]
+== Additional resources
+* xref:../../support/index.adoc#support-overview[Support overview]

ai_workloads/kueue/images

@@ -0,0 +1 @@
+../../images/