Skip to content

Commit 889a4d4

Browse files
Nikhil-Ladhamergify[bot]
Nikhil-Ladha
authored and
mergify[bot]
committed
docs: add design doc for VR, VGR
add design doc for VR, VGR that can be referenced by users to understand the workflow for them in csi-addons Signed-off-by: Nikhil-Ladha <nikhilladha1999@gmail.com>
1 parent 31bf725 commit 889a4d4

File tree

4 files changed

+233
-0
lines changed

4 files changed

+233
-0
lines changed

‎docs/design/volumegroupreplication.md‎

Lines changed: 133 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,133 @@
1+
# Summary
2+
3+
This document discusses the csi-addons API that allows replication of volume groups to a secondary cluster.
4+
5+
## Terminologies
6+
7+
For the sake of simplicity, we will be using some abbreviations in the document which are as follows:
8+
9+
- DR - Disaster Recovery
10+
- K8s - Kubernetes
11+
- PVC - Persistent Volume Claim
12+
- PV - Persistent Volume
13+
- Primary - The cluster where the application workloads are actively running add data is written to. It initiates the replication to the secondary cluster.
14+
- Secondary - The cluster that receives replicated data from the primary cluster.
15+
- SC - Storage Class
16+
- VGR - VolumeGroupReplication
17+
- VGRC - VolumeGroupReplicationClass
18+
- VR - VolumeReplication
19+
- VRC - VolumeReplicationClass
20+
21+
## Motivation
22+
23+
Most of the storage vendors provide a mechanism to replicate data to other nodes, clusters, peers. But, there are no K8s APIs that provide replication of a group of volumes at real time. We have K8s VolumeGroupSnapshot APIs provided but that again involves adding another application to take backup and restore it on the secondary cluster.
24+
Therefore, there was a need to have K8s APIs that represents group of volumes and replicate the group of volumes at real time to provide support for Disaster Recovery (DR).
25+
26+
## Goal
27+
28+
- Provide APIs to support replication of a group of volumes.
29+
30+
## Design for VolumeGroupReplication
31+
32+
This section discusses the API definition, implementation nuances of VolumeGroupReplication (VGR), VolumeGroupReplicationContent (VGRContent) and VolumeGroupReplicationClass (VGRC).
33+
34+
VolumeGroupReplication (VGR) is a namespaced scope resource that contains references to the PVCs that are to be grouped and replicated to a secondary cluster.
35+
36+
VolumeGroupReplicationContent (VGRContent) is a clusterscoped resource that contains volume grouping related information and reference to the PVs part of the group.
37+
38+
VolumeGroupReplicationClass (VGRC) is a clusterscoped resource that contains driver related configuration parameters required for group replication. It is similar to the definition of a StorageClass (SC) in K8s but crafted for VolumeGroupReplication.
39+
40+
### Implementation/Workflow Details
41+
42+
![VolumeGroupReplication Workflow Diagram](volumegroupreplication_arch.svg)
43+
44+
To start with, admin should create the VRC, VGRC which contains the provisioner (csi driver) that supports replication capability and the volumes (PVCs) should be created using the same provisioner. Also, add the necessary parameters required by the storage vendor to mirror the group to a secondary cluster like the secret name, namespace.
45+
46+
To start replication of the group, the user needs to create a VGR that contains the PVC label selector as the source which would be used to filter the PVCs that are supposed to be part of the same group and replicated to a secondary cluster, along with other required fields like the VRC (created above), VGRC (created above), and the desired replicationState of the VGR.
47+
The VGR only groups PVCs that are part of the same namespace as that of VGR.
48+
The `replicationState` indicates the current role of the VGR such as `primary` means the VGR is in the active replication role, sending data to a secondary cluster, or `secondary` that means the VGR is in the passive role, receiving replicated data from the primary cluster (having VGR is not manadatory here when its in passive role).
49+
50+
VGR’s spec consists of a field named external which signifies if the CR should be reconciled and managed by the csi-addons volumegroupreplication_controller or by the storage vendor specific implementation of the VGR controller.
51+
When set to true the CR won’t be reconciled by the csi-addons “volumegroupreplication_controller”, and the storage vendor is expected to deploy a controller that manages the CR. The value is set to `false` by default.
52+
53+
When a vendor implements their storage specific controller to manage the VGR using the external field, it is the responsibility of the operator/controller watching the VGR to add the necessary validations and issue the required RPCs to enable mirroring on the group at the storage level.
54+
The VGR creation flow involves creating a VGRContent and a VR as well respectively for it. VGRContent is a cluster scoped CR that is created per VGR. It contains a reference to the VGR in its spec that is used to identify the VGR that manages it.
55+
56+
When a VGR is created, the controller should list all the PVCs that match the label selector and can be a part of the volume group. The controller should then be creating a VGRContent for the VGR that manages the group level operations like creating, modifying, deleting etc, which can again be delegated to another controller that manages the VGRContent.
57+
Once the group is created the VGRContent should be updated with the `volumeGroupReplicationHandle` that can be checked by the VGR controller to go ahead and create the VR which would manage the mirroring based operations.
58+
The VGRContent’s Status has to be populated with `volumeHandles` of all the PV’s that are part of the group, where these PVs belong to the PVCs which are filtered using the source in VGR’s Spec.And, in a similar way the VGR’s Status.persistentVolumeClaimsRefList field should be updated with the names of all the PVCs that are part of the group.
59+
60+
In other words, the VGR controller shouldn’t be managing the group managing and mirroring operations on its own, instead it can make use of the existing VR controller which manages mirroring operations of a volume and delegate the group operations to a separate VGRContent controller.
61+
62+
For reference, one can check the implementation of csi-addons controllers for the same:
63+
64+
- [volumereplication_controller](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/internal/controller/replication.storage/volumereplication_controller.go) - It manages the mirroring operations for a volume/group.
65+
- [volumegroupreplication_controller](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/internal/controller/replication.storage/volumegroupreplication_controller.go) - It manages the flow of VGR and necessary validations to keep the VGR as the source of truth for the user.
66+
- [volumegroupreplicationcontent_controller](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/internal/controller/replication.storage/volumegroupreplicationcontent_controller.go) - It manages the group level operations for the VGR.
67+
68+
VGR has a Status field that is basically the same as that of VolumeReplication’s Status field and it can be utilized by the storage vendor to populate the current status of replication to the end user and report any errors as such at the CR level.
69+
70+
### Pre-Provisioned VolumeGroup
71+
72+
Admin can create a VGRContent specifying an existing `volumeGroupReplicationHandle` in the storage system and specifying an existing VolumeGroup name and namespace. Then, the user can create a VolumeGroup that points to the VGRContent name.
73+
74+
The VGR controller will filter all the PVCs that match the label selector of the VGR, and the VGRContent controller will issue a Modify RPC call to update the group with any new PVC, or shall return if the group is unchanged. Then, the VGR controller can manage mirroring on the volume group and enable mirroring on the group.
75+
76+
### Delete VolumeGroupReplication
77+
78+
When a VolumeGroupReplication is deleted the controller should first remove the volumes from the group, and then delete the group. Or, it can choose to disable mirroring and then remove the volumes from the group as well.
79+
Then, delete the VolumeGroup from the storage system. And, delete the VGRContent and perform deletion of VR which would disable the mirroring operations, remove any sanitary annotations from PVCs, VGR if any. Then, at the end the VolumeGroup CR should be deleted by the controller.
80+
81+
## API definition of VolumeGroupReplication (VGR) CRD
82+
83+
- [VR Type Definition](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/api/replication.storage/v1alpha1/volumegroupreplication_types.go)
84+
- [VR CRD Definition](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/config/crd/bases/replication.storage.openshift.io_volumegroupreplications.yaml)
85+
86+
## API definition of VolumeGroupReplicationContent CRD
87+
88+
- [VR Type Definition](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/api/replication.storage/v1alpha1/volumegroupreplicationcontent_types.go)
89+
- [VR CRD Definition](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/config/crd/bases/replication.storage.openshift.io_volumegroupreplicationcontents.yaml)
90+
91+
## API definition of VolumeGroupReplicationClass (VGRC) CRD
92+
93+
- [VR Type Definition](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/api/replication.storage/v1alpha1/volumegroupreplicationclass_types.go)
94+
- [VR CRD Definition](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/config/crd/bases/replication.storage.openshift.io_volumegroupreplicationclasses.yaml)
95+
96+
## Example YAML files
97+
98+
- VolumeGroupReplicationClass (VGRC) CR:
99+
100+
```yaml
101+
apiVersion: replication.storage.openshift.io/v1alpha1
102+
kind: VolumeGroupReplicationClass
103+
metadata:
104+
name: volumegroupreplicationclass-sample
105+
spec:
106+
provisioner: example.provisioner.io
107+
parameters:
108+
clusterID: my-cluster
109+
replication.storage.openshift.io/group-replication-secret-name: secret-name
110+
replication.storage.openshift.io/group-replication-secret-namespace: secret-namespace
111+
```
112+
113+
- VolumeGroupReplication (VGR) CR:
114+
115+
```yaml
116+
apiVersion: replication.storage.openshift.io/v1alpha1
117+
kind: VolumeGroupReplication
118+
metadata:
119+
name: volumegroupreplication-sample
120+
namespace: default
121+
spec:
122+
volumeReplicationClassName: volumereplicationclass-sample
123+
volumeGroupReplicationClassName: volumegroupreplicationclass-sample
124+
replicationState: primary
125+
source:
126+
selector:
127+
matchLabels:
128+
appname: test
129+
status:
130+
persistentVolumeClaimsRefList:
131+
- test-pvc
132+
state: primary
133+
```

‎docs/design/volumegroupreplication_arch.svg‎

Lines changed: 5 additions & 0 deletions
Loading

‎docs/design/volumereplication.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Summary
2+
3+
This document discusses the csi-addons API that allows replication of volumes to another peer cluster.
4+
5+
## Terminologies
6+
7+
For the sake of simplicity, we will be using some abbreviations in the document which are as follows:
8+
9+
- VR - VolumeReplication
10+
- VRC - VolumeReplicationClass
11+
- K8s - Kubernetes
12+
- DR - Disaster Recovery
13+
- PVC - Persistent Volume Claim
14+
- PV - Persistent Volume
15+
- Primary - Cluster where the applications are currently running
16+
- Secondary - Cluster where the application data is being replicated
17+
18+
## Motivation
19+
20+
Most of the storage vendors provide a mechanism to replicate data to other nodes, clusters, peers in order to achieve crash consistency for the data stored on the volumes. But, there are no K8s APIs that provide crash consistency at real time. We have K8s VolumeGroupSnapshot APIs provided but that again involves adding another application to take backup and restore it on the secondary cluster.
21+
Therefore, there was a need to have K8s APIs that provide replication of volumes and a group of volumes at real time to provide support for Disaster Recovery (DR).
22+
23+
## Goal
24+
25+
- Provide APIs to support replication of a group of volumes.
26+
27+
## Design for VolumeReplication
28+
29+
This section discusses the API definition, implementation nuances of VolumeReplication (VR) and VolumeReplicationClass (VRC).
30+
31+
VolumeReplication is a namespaced scope resource that contains reference to a storage object (a PVC) and VolumeReplicationClass corresponding to the driver providing replication. VR contains the details that explains which PVC is being replicated and what’s the current status of replication along with errors, if any.
32+
33+
VolumeReplicationClass (VRC) is a cluster scoped resource that contains driver related configuration parameters required for replication. It is similar to the definition of a StorageClass (SC) in K8s but crafted for VolumeReplication.
34+
35+
### Implementation/Workflow Details
36+
37+
![VolumeReplication Workflow Diagram](volumereplication_arch.svg)
38+
39+
To start with, users should create the VRC which contains the provisioner (csi driver) that supports mirroring capability and the volumes (PVCs) should be created using the same provisioner. Also, add the necessary parameters required by the storage vendor to mirror the volume to a peer cluster like the secret name, namespace.
40+
41+
To start replication of the volume, the user needs to create a VR that contains the PVC as the dataSource which would be mirrored/replicated to a peer cluster, along with other required fields like the volumeReplicationClass (shared below) and the desired replicationState of the VR.
42+
The VR supports PVC and VolumeGroupReplication as the `dataSource` and the storage vendor is responsible for adding necessary checks to validate that if the dataSource is a single volume or a volumegroup and proceed accordingly. `replicationState` is used to determine if the VR is replicating to a secondary cluster i.e. `primary` or is on replicating end of the mirroring i.e. `secondary`.
43+
44+
## API definition of VolumeReplication (VR) CRD
45+
46+
- [VR Type Definition](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/api/replication.storage/v1alpha1/volumereplication_types.go)
47+
- [VR CRD Definition](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/config/crd/bases/replication.storage.openshift.io_volumereplications.yaml)
48+
49+
## API definition of VolumeReplicationClass (VR) CRD
50+
51+
- [VRClass Type Definition](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/api/replication.storage/v1alpha1/volumereplicationclass_types.go)
52+
- [VRClass CRD Definition](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/config/crd/bases/replication.storage.openshift.io_volumereplicationclasses.yaml)
53+
54+
## Example YAML files
55+
56+
- VolumeReplicationClass CR:
57+
58+
```yaml
59+
apiVersion: replication.storage.openshift.io/v1alpha1
60+
kind: VolumeReplicationClass
61+
metadata:
62+
  name: volumereplicationclass-sample
63+
spec:
64+
  provisioner: example.provisioner.io
65+
  parameters:
66+
    replication.storage.openshift.io/replication-secret-name: secret-name
67+
    replication.storage.openshift.io/replication-secret-namespace: secret-namespace
68+
    # schedulingInterval is a vendor specific parameter. It is used to set the
69+
    # replication scheduling interval for storage volumes that are replication
70+
    # enabled using related VolumeReplication resource
71+
    schedulingInterval: 1m
72+
```
73+
74+
- VolumeReplication CR:
75+
76+
```yaml
77+
apiVersion: replication.storage.openshift.io/v1alpha1
78+
kind: VolumeReplication
79+
metadata:
80+
  name: volumereplication-sample
81+
  namespace: default
82+
spec:
83+
  volumeReplicationClass: volumereplicationclass-sample
84+
  replicationState: primary
85+
  replicationHandle: replicationHandle # optional
86+
  dataSource:
87+
    kind: PersistentVolumeClaim
88+
    name: myPersistentVolumeClaim # should be in same namespace as VolumeReplication
89+
```

0 commit comments

Comments
 (0)