Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
080b811
initial support for using non couchbase user through sudo
Feb 23, 2021
d43b841
Merge branch 'add_noncouchbase' into develop
Feb 23, 2021
8ed6f50
issue #22 - initial deployment, index support changes for backup inge…
Mar 2, 2021
92ae6aa
Merge branch 'issue#22' into develop
Mar 2, 2021
8dffc10
partial work
Mar 4, 2021
012b1b2
Merge branch 'couchbase_ce' into develop
Mar 4, 2021
b1693c9
cbbackupmgr working with sudo, bucket list in snapshot in JSON
Mar 5, 2021
2d9f697
Merge branch 'cbmgrcouchbaseplusbucket' into develop
Mar 5, 2021
d7aaa2e
version 1.0.2
Mar 9, 2021
265037d
fixing uid/gid regex
Mar 9, 2021
8add5fe
version 1.0.3 - error messages fix
Mar 11, 2021
d5ab266
ver 1.0.3.1
Mar 16, 2021
2b84ef9
ver 1.0.3.2 - sudo for all
Mar 16, 2021
4345c63
resync of xdcr and backupmgr OK
Jun 22, 2021
0d7d181
workinf with copy config
Jul 4, 2021
d0a5216
working with indexes - tested with backup only - ee, analytics remove…
Jul 5, 2021
429d3f5
Merge branch 'cloneconfig' into develop
Jul 6, 2021
9156c99
add end of multi cluster
Jul 9, 2021
9c9420f
working multinode
Jul 13, 2021
e50edbe
Merge branch 'multinode' into develop
Jul 14, 2021
78b6581
checks added, upgrade added
Jul 21, 2021
0d1deba
Merge branch 'cleanup' into develop
Jul 21, 2021
5e6a722
version 1.1.0 build 0
Jul 22, 2021
ef46527
check mount points / doc updates
Aug 6, 2021
0e8165f
cbbackup exit code check
Sep 2, 2021
7179322
xdcr fixed for multi IP, and different scenarios
Sep 7, 2021
caca26e
v1.1.1 build 1.1.0.2
Sep 7, 2021
83e1cc4
v1.1.0 build 1.1.0.2
Sep 7, 2021
30c574d
version 1.1.1 - added field for hostname
Oct 7, 2021
e064819
Merge branch 'xdcr_fixes' into develop
Oct 7, 2021
f863d04
version 1.1.1
Oct 28, 2021
1217f8f
Issue #32 - multinode start
Dec 9, 2021
c9e0625
version 1.1.2
Dec 9, 2021
3062019
Set theme jekyll-theme-minimal
pioro Jan 7, 2022
3fcd4e1
doc fixes
Jan 11, 2022
8ed4f32
Merge branch 'master' of github.com:pioro/couchbase-plugin
Jan 11, 2022
c3c18a4
new logo
Jan 11, 2022
2eae51e
proper style
Jan 11, 2022
d60e9a6
docs with multi node
Jan 17, 2022
160d245
Troubleshooting added
Feb 16, 2022
7de348c
Troubleshooting added
Feb 16, 2022
ca8b09f
Merge branch 'select-connector' into master
ayesharustgi-delphix Jul 15, 2022
e2f8748
1.1.2.0
Jul 18, 2022
64ef4a6
Merge branch 'master' of github.com:pioro/couchbase-plugin
Jul 18, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/_config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
theme: jekyll-theme-minimal
2 changes: 1 addition & 1 deletion docs/docs/.pages
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
arrange:
- Overview: 'index.md'
- [Installation]: 'Installation.md'
- PluginDevelopment : 'PluginDevelopment.md'
- Discovery: 'Discovery.md'
- Linking: 'Linking.md'
- Provisioning: 'Provisioning.md'

68 changes: 34 additions & 34 deletions docs/docs/Discovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,27 @@ Environment discovery is a process that enables the Couchbase Plugin to determin
. Whenever there is any change (installing a new database home) to an already set up environment in the Delphix application, we need to perform an environment refresh.


Prerequisites
=============
## Prerequisites

- A source environment must be added to the Delphix Engine.
- A stagine or target environment must be added to the Delphix Engine.
- Installation of the Couchbase Plugin is required before the Discovery.
- Environment variable `$COUCHBASE_PATH ` should set on staging/target host, which contains the binary path of Couchbase.


Refreshing an Environment
=========================
Environment refresh will update the metadata associated with that environment and send a new Plugin to the host.
### Configuring a staging environment

1. Login to the **Delphix Management** application.
2. Click **Manage**.
3. Select **Environments**.
4. In the Environments panel, click the name of the environment you want to refresh.
5. Select the **Refresh** icon.
6. In the Refresh confirmation dialog select **Refresh**.
Environments contain repositories, and each environment may have any number of repositories associated with it.
Couchbase is limiting number of Couchbase binaries installed on server to one, and in that case each environment will contain
a single repository with current Couchbase installation.

Repository contains database instances and in each repository any number of source cluster can be configured.
Please keep in mind that only one dSource can be enabled simiultaniusly on a given staging server.

![Screenshot](/couchbase-plugin/image/image9.png)
For next step go to XDCR Setup or Backup Setup sections below

### XDCR Setup

XDCR Setup
===================
Environments exist to contain `repositories`, and each environment may have any number of repositories associated with it.
`Repository` contains database instances and in each repository any number of `SourceConfig` objects, which represent known database instances. Source config is not generated automatically in
Couchbase plugin. Therefore, we need to add `SourceConfig` object through which can create a dSource.
By default Couchbase cluster are not discovered and has to be added manually using the following steps:

1. Login to the **Delphix Management** application.
2. Click **Manage**.
Expand All @@ -40,40 +34,46 @@ Environments exist to contain `repositories`, and each environment may have any
5. Click on **+** icon (Shown in next image).


![Screenshot](/couchbase-plugin/image/image10.png)
![Screenshot](./image/add_repository.png)


6. Add required details in the `Add database` section.
- Enter port number in **Source Couchbase port** section.
- Enter source host address in section **Source Host**.
- Enter source host address in section **Source Host** ( this will be used as Source cluster for XDCR replication )
- Enter unique name for the staging database in **identify field** section.
- Enter Couchbase data path of staging host in **DB data path** section.


![Screenshot](/couchbase-plugin/image/image11.png)

![Screenshot](./image/add_sourceconfig_xdcr.png)

CBBACKUPMGR Setup
=================
### Backup Setup

The steps to add source config remain the same as we saw in XDCR setup. In this approach, we don't connect to source environment as this is zero-touch production approach.
We can enter any random or dummy value in this field of source host name when we choose CBBACKUPMGR option for data ingestion.
By default Couchbase clusters are not discovered and has to be added manually using the following steps:

1. Login to the **Delphix Management** application.
2. Click **Manage**.
3. Select **Environments**.
4. Select the repository.
5. Click on **+** icon (Shown in next image).
![Screenshot](/couchbase-plugin/image/image10.png)
5. Click on **+** icon (Shown in next image).

6. In the **Add Database** section enter the following information:
- `Source Couchbase port`: This is the port number to be used by Couchbase services.
- `Source Host`: Leave this field as blank.
- `identity field`: Provide unique name for staging database.
- `DB data path`: Leave this field as blank.
![Screenshot](./image/add_repository.png)

6. Add required details in the `Add database` section.
- Enter port number in **Source Couchbase port** section ( this can be any dummy number - this field is not used for backup based ingestion )
- Enter source host address in section **Source Host** ( this can be any dummy number - this field is not used for backup based ingestion )
- Enter unique name for the staging database in **identify field** section.
- Enter Couchbase data path of staging host in **DB data path** section.

![Screenshot](/couchbase-plugin/image/image11.png)

## Refreshing an Environment
Environment refresh will update the metadata associated with that environment and send a new Plugin to the host.

1. Login to the **Delphix Management** application.
2. Click **Manage**.
3. Select **Environments**.
4. In the Environments panel, click the name of the environment you want to refresh.
5. Select the **Refresh** icon.
6. In the Refresh confirmation dialog select **Refresh**.

![Screenshot](./image/add_sourceconfig_backup.png)

12 changes: 6 additions & 6 deletions docs/docs/Installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,22 +14,22 @@ Method1: Using GUI
------------------

1. Click on **Manage** (present on top-left of this page) and then **Plugins**.
![Screenshot](/couchbase-plugin/image/image2.png)
![Screenshot](./image/image2.png)

2. Click on **+** icon.
![Screenshot](/couchbase-plugin/image/image3.png)
![Screenshot](./image/image3.png)

3. Click on **Upload** a plugin.
![Screenshot](/couchbase-plugin/image/image4.png)
![Screenshot](./image/image4.png)

4. Select the `build(artifacts.json)` from your device.
![Screenshot](/couchbase-plugin/image/image5.png)
![Screenshot](./image/image5.png)

5. Click on **close** button.
![Screenshot](/couchbase-plugin/image/image6.png)
![Screenshot](./image/image6.png)

6. See the plugin version in **Plugins** section.
![Screenshot](/couchbase-plugin/image/image7.png)
![Screenshot](./image/image7.png)


Method2: Using dvp command
Expand Down
156 changes: 103 additions & 53 deletions docs/docs/Linking.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,90 +4,140 @@ Linking a data source will create a dSource object on the engine and allow Delph



Prerequisites
=============
## Prerequisites

Discovery and source config object should be created on the staging host before proceeding to link Couchbase dataset.


Creating dSource
==============
Staging environment created and source database configured in discovered repository

## Creating dSource using XDCR

1. Login to **Delphix Management** application.
2. Click **Manage** > **Datasets**.
3. Select **Add dSource**.
4. In the Add dSource wizard, select the Couchbase source configuration which is created on the staging host.
5. Enter the Couchbase-specific parameters for your dSource configuration.
6. Select the dSource approach from the drop-down (XDCR and Couchbase Backup Manager) available on dSource wizard.
6. Select the dSource type XDCR from the drop-down available on dSource wizard.
7. Based on approach selection, follow the steps either for XDCR or Couchbase Backup Manager method. The Description of both methods is below.


Method1: XDCR
--------------
Cross datacenter replication allows data to be replicated across clusters that are potentially located in different data centers.


1. Enter the details for **Staging Couchbase host** - FQDN or IP address recommended.
2. Enter the details for **Staging Port Number** available on the staging host. The default port for couchbase is 8091.
3. Enter the details for **Mount Path** available on the staging host. This empty folder acts as a base for NFS mounts.
4. Enter the details for **Staging Cluster Name** to setup new cluster on the staging host.
5. Enter the configuration details for your staging cluster as per resource availability on the staging host.
8. Enter the details for **Staging Couchbase host** - FQDN or IP address recommended.
9. Enter the details for **Staging Port Number** available on the staging host. The default port for couchbase is 8091.
10. Enter the details for **Mount Path** available on the staging host. This empty folder acts as a base for NFS mounts.
11. Enter the details for **Staging Cluster Name** to setup new cluster on the staging host.
12. Enter the configuration details for your staging cluster as per resource availability on the staging host.
- Cluster RAM Size
- Cluster Index RAM Size
- Cluster FTS RAM Size
- Cluster Eventing RAM Size
- Cluster Analysis RAM Size
![Screenshot](/couchbase-plugin/image/image12.png)
- Cluster Eventing RAM Size - this should be 0
- Cluster Analysis RAM Size - this should be 0

6. Click on **+** plus symbol to modify configuration settings. Mention bucket list for which cross datacenter replication (XDCR) only be enabled.
![Screenshot](/couchbase-plugin/image/image14.png)
![Screenshot](./image/add_dsource_1.png)

7. Enter the details of **Bucket Name** to be part of XDCR. Then click on **Next** button
![Screenshot](/couchbase-plugin/image/image15.png)
13. Enter the details for **Staging Cluster Admin User** and **Staging Cluster Admin Password**
14. Enter the details for **Source Cluster Admin User** and **Source Cluster Admin Password**

8. Provide the details for **dSource Name** and **Target group** on the dSource configuration page.
![Screenshot](/couchbase-plugin/image/image16.png)
![Screenshot](./image/add_dsource_2.png)

9. On the **Data management** page, select the following:
15. If not all buckets needs to be replicated, click on **+** plus symbol to modify configuration settings. Mention bucket list for which cross datacenter replication (XDCR) only be enabled.
![Screenshot](./image/image14.png)

16. Enter the details of **Bucket Name** to be part of XDCR. Then click on **Next** button
![Screenshot](./image/image15.png)

17. Provide the details for **dSource Name** and **Target group** on the dSource configuration page.
![Screenshot](./image/add_dsource_3.png)

18. On the **Data management** page, select the following:
- Staging Environment: This will be your staging host where source config was created.
- User: Database OS user with required privileges for linking the dataset.
10. On the next section, review the configuration and click on **Next** button to view the summary.
11. Click the **Submit** button which will initiate the linking process.
![Screenshot](/couchbase-plugin/image/image17.png)
12. Once dSource is created successfully, you can review the datasets on **Manage** > **Datasets** > **dSource Name**.
![Screenshot](/couchbase-plugin/image/image19.png)

![Screenshot](./image/add_dsource_4.png)

19. On the next screens, configure a policy, hooks and review the configuration and click on **Next** button to view the summary.

![Screenshot](./image/add_dsource_5.png)

Method2: Couchbase Backup Manager
---------------------------------------
**Note**: Follow the instructions below before creating dSource to avoid source/production server dependency.
![Screenshot](./image/add_dsource_6.png)

- Provide source server buckets related information in a file: */tmp/couchbase_src_bucket_info.cfg*.
`/opt/couchbase/bin/couchbase-cli bucket-list --cluster <sourcehost>:8091 --username $username --password $password`

- **Backup Repository**: This file will be required at the time of dSource creation using CBBACKUPMGR.
`/opt/couchbase/bin/cbbackupmgr config --archive /u01/couchbase_backup --repo delphix`
20. Click the **Submit** button which will initiate the linking process.

- **Backup Location**: Get data from source host in backup directory of staging host.
`/opt/couchbase/bin/cbbackupmgr backup -a /u01/couchbase_backup -r delphix -c couchbase://<hostname> -u user -p password`
![Screenshot](./image/add_dsource_7.png)


**Procedure**:
21. Once dSource is created successfully, you can review the datasets on **Manage** > **Datasets** > **dSource Name**.


![Screenshot](./image/dsource_ingested.png)


## Creating dSource using backup

Prerequisites:
Access to production backup with those ex. values:

/backup - archive

PROD - repository

```
/opt/couchbase/bin/cbbackupmgr info -a /backup -r PROD
Name | Size | # Backups |
PROD | 58.03MB | 1 |
+ Backup | Size | Type | Source | Cluster UUID | Range | Events | Aliases | Complete |
+ 2022-01-10T11_29_26.465860528-05_00 | 58.03MB | FULL | http://couchbasesrc.dlpxdc.co:8091 | 08f7937a26b2d20178a5ed16d7a2dd1c | N/A | 0 | 0 | true |
```


1. Login to **Delphix Management** application.
2. Click **Manage** > **Datasets**.
3. Select **Add dSource**.
4. In the **Add dSource wizard**, select the Couchbase source configuration you created on the staging host.
4. In the Add dSource wizard, select the Couchbase source configuration which is created on the staging host.
5. Enter the Couchbase-specific parameters for your dSource configuration.
6. Select the dSource type from the drop-down available on dSource wizard.
7. When we select CBBACKUPMGR as dSource Type, the following fields on dSource wizard are mandatory.
- Enter the details for **Backup Location** where the backup files generated through CBBACKUPMGR are present on the staging host.
- Enter the details for **Backup Repository** that contains a backup configuration of staging host.
8. The remaining steps for CBBACKUPMGR ingestion are similar to XDCR. Use steps from the second point mentioned in XDCR method.
6. Select the dSource type Couchbase Backup Manager from the drop-down available on dSource wizard.
7. Based on approach selection, follow the steps either for XDCR or Couchbase Backup Manager method. The Description of both methods is below.
8. Enter the details for **Staging Couchbase host** - FQDN or IP address recommended.
9. Enter the details for **Staging Port Number** available on the staging host. The default port for couchbase is 8091.
10. Enter the details for **Backup Location** available on the staging host.
11. Enter the details for **Backup repository**
12. Enter the details for **Mount Path** available on the staging host. This empty folder acts as a base for NFS mounts.
13. Enter the details for **Staging Cluster Name** to setup new cluster on the staging host.
14. Enter the configuration details for your staging cluster as per resource availability on the staging host.
- Cluster RAM Size
- Cluster Index RAM Size
- Cluster FTS RAM Size
- Cluster Eventing RAM Size - this should be 0
- Cluster Analysis RAM Size - this should be 0

![Screenshot](./image/add_dsource_1backup.png)

15. Enter the details for **Staging Cluster Admin User** and **Staging Cluster Admin Password**
16. Enter dummy values for **Source Cluster Admin User** and **Source Cluster Admin Password** - they are not used

![Screenshot](./image/add_dsource_2backup.png)

17. Then click on **Next** button

17. Provide the details for **dSource Name** and **Target group** on the dSource configuration page.
![Screenshot](./image/add_dsource_3.png)

18. On the **Data management** page, select the following:
- Staging Environment: This will be your staging host where source config was created.
- User: Database OS user with required privileges for linking the dataset.

![Screenshot](./image/add_dsource_4.png)

19. On the next screens, configure a policy, hooks and review the configuration and click on **Next** button to view the summary.

![Screenshot](./image/add_dsource_5.png)

![Screenshot](./image/add_dsource_6.png)


20. Click the **Submit** button which will initiate the linking process.

![Screenshot](./image/add_dsource_7backup.png)


21. Once dSource is created successfully, you can review the datasets on **Manage** > **Datasets** > **dSource Name**.

Note: When we select dSource type as Couchbase Backup Manager, we do not require any details for the `Staging Couchbase Host` field.

![Screenshot](/couchbase-plugin/image/image22.png)
![Screenshot](./image/dsource_ingested.png)

Loading