|
| 1 | +# Demo Outline |
| 2 | +:white_check_mark: Setup and start Gemfire database (Source)<br> |
| 3 | +:white_check_mark: Setup and start Redis Enterprise database (Target)<br> |
| 4 | +:white_check_mark: Setup and start Redis Connect<br> |
| 5 | +:white_check_mark: Perform Initial load and CDC with Redis Connect<br> |
| 6 | + |
| 7 | +# Prerequisites |
| 8 | +Docker compatible [*nix OS](https://en.wikipedia.org/wiki/Unix-like) and [Docker](https://docs.docker.com/get-docker) installed. |
| 9 | +<br>Please have 8 vCPU*, 8GB RAM and 50GB storage for this demo to function properly. Adjust the resources based on your requirements. For HA, at least have 2 Redis Connect instances deployed on separate hosts.</br> |
| 10 | +<br>Execute the following commands (copy & paste) to download and setup Redis Connect and demo scripts. |
| 11 | +i.e.</br> |
| 12 | + |
| 13 | +```bash |
| 14 | +wget -c https://github.com/redis-field-engineering/redis-connect-dist/archive/main.zip && \ |
| 15 | +mkdir -p redis-connect/demo && \ |
| 16 | +unzip main.zip "redis-connect-dist-main/examples/gemfire/*" -d redis-connect && \ |
| 17 | +cp -R redis-connect/redis-connect-dist-main/examples/gemfire/demo/* redis-connect/demo && \ |
| 18 | +rm -rf main.zip redis-connect/redis-connect-dist-main && \ |
| 19 | +cd redis-connect && \ |
| 20 | +chmod a+x demo/*.sh && \ |
| 21 | +cd demo |
| 22 | +``` |
| 23 | + |
| 24 | +## Setup Gemfire database (Source) |
| 25 | +<b>_PostgreSQL on Docker_</b> |
| 26 | +<br>Execute [setup_gemfire.sh](setup_gemfire.sh)</br> |
| 27 | +```bash |
| 28 | +demo$ ./setup_gemfire.sh |
| 29 | +``` |
| 30 | + |
| 31 | +<details><summary>Validate Gemfire database is running as expected:</summary> |
| 32 | +<p> |
| 33 | + |
| 34 | +```bash |
| 35 | +demo$ sudo docker ps -a | grep gemfire |
| 36 | +beb0205a037f apachegeode/geode:1.12.9 "sh -c /geode/script…" 4 hours ago Up 4 hours 0.0.0.0:1099->1099/tcp, 0.0.0.0:7070->7070/tcp, 0.0.0.0:8080->8080/tcp, 0.0.0.0:10334->10334/tcp, 0.0.0.0:40404->40404/tcp gemfire-1.12.9-virag-WPG6PH3FV5 |
| 37 | + |
| 38 | +demo$ sudo docker exec -it gemfire-1.12.9-$(hostname) sh -c "gfsh version" |
| 39 | +1.12.9 |
| 40 | +``` |
| 41 | +</p> |
| 42 | +</details> |
| 43 | + |
| 44 | +## Setup Redis Enterprise cluster, databases and RedisInsight in docker (Target) |
| 45 | +<br>Execute [setup_re.sh](setup_re.sh)</br> |
| 46 | +```bash |
| 47 | +demo$ ./setup_re.sh |
| 48 | +``` |
| 49 | +**NOTE** |
| 50 | + |
| 51 | +The above script will create a 1-node Redis Enterprise cluster in a docker container, [Create a target database with RediSearch module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a job management and metrics database with RedisTimeSeries module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a RediSearch index for emp Hash](https://redislabs.com/blog/getting-started-with-redisearch-2-0/), [Start a docker instance of grafana with Redis Data Source](https://redisgrafana.github.io/) and [Start an instance of RedisInsight](https://docs.redislabs.com/latest/ri/installing/install-docker/). |
| 52 | + |
| 53 | +## Start Redis Connect |
| 54 | + |
| 55 | +<details><summary>Review options by running Redis Connect docker container </summary> |
| 56 | +<p> |
| 57 | + |
| 58 | +```bash |
| 59 | +demo$ docker run \ |
| 60 | +-it --rm --privileged=true \ |
| 61 | +--name redis-connect-$(hostname) \ |
| 62 | +-v $(pwd)/config:/opt/redislabs/redis-connect/config \ |
| 63 | +-v $(pwd)/config/samples/credentials:/opt/redislabs/redis-connect/config/samples/credentials \ |
| 64 | +--net host \ |
| 65 | +redislabs/redis-connect |
| 66 | +``` |
| 67 | + |
| 68 | +</p> |
| 69 | +</details> |
| 70 | + |
| 71 | +<details><summary>Expected output:</summary> |
| 72 | +<p> |
| 73 | + |
| 74 | +```bash |
| 75 | +------------------------------- |
| 76 | +Redis Connect startup script. |
| 77 | +******************************* |
| 78 | +Please ensure that these environment variables are correctly mapped before executing start and cli options. They can also be found in /opt/redislabs/redis-connect/bin/redisconnect.conf |
| 79 | +Example environment variables and volume mapping for docker based deployments |
| 80 | +-e REDISCONNECT_JOB_MANAGER_CONFIG_PATH=/opt/redislabs/redis-connect/config/jobmanager.properties [OPTIONAL] |
| 81 | +-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect/config/logback.xml [OPTIONAL] |
| 82 | +-e REDISCONNECT_JAVA_OPTIONS=-Xms1g -Xmx2g [OPTIONAL] |
| 83 | +-e REDISCONNECT_EXTLIB_DIR=/opt/redislabs/redis-connect/extlib [OPTIONAL] |
| 84 | +-v <HOST_PATH_TO_JOB_MANAGER_PROPERTIES>:/opt/redislabs/redis-connect/config |
| 85 | +-v <HOST_PATH_TO_CREDENTIALS>:/opt/redislabs/redis-connect/config/samples/credentials |
| 86 | +-v <HOST_PATH_TO_EXTLIB>:/opt/redislabs/redis-connect/extlib [OPTIONAL] |
| 87 | +-p 8282:8282 |
| 88 | + |
| 89 | +Usage: [-h|cli|start] |
| 90 | +options: |
| 91 | +-h: Print this help message and exit. |
| 92 | +-v: Print version. |
| 93 | +cli: init Redis Connect CLI |
| 94 | +start: init Redis Connect Instance (Cluster Member) |
| 95 | +------------------------------- |
| 96 | +``` |
| 97 | + |
| 98 | +</p> |
| 99 | +</details> |
| 100 | + |
| 101 | +<details><summary><b>Start Redis Connect Instance</b></summary> |
| 102 | +<p> |
| 103 | + |
| 104 | +```bash |
| 105 | +docker run \ |
| 106 | +-it --rm --privileged=true \ |
| 107 | +--name redis-connect-$(hostname) \ |
| 108 | +-v $(pwd)/config:/opt/redislabs/redis-connect/config \ |
| 109 | +-v $(pwd)/extlib:/opt/redislabs/redis-connect/extlib \ |
| 110 | +--net host \ |
| 111 | +redislabs/redis-connect start |
| 112 | +``` |
| 113 | + |
| 114 | +</p> |
| 115 | +</details> |
| 116 | + |
| 117 | +<details><summary>Expected output:</summary> |
| 118 | +<p> |
| 119 | + |
| 120 | +```bash |
| 121 | +``` |
| 122 | + |
| 123 | +</p> |
| 124 | +</details> |
| 125 | + |
| 126 | +**Open browser to access Swagger UI -** [http://localhost:8282/swagger-ui/index.html]() |
| 127 | +<br>_For quick start, use '**cdc_job**' as **jobName**_ |
| 128 | +<br><br><img src="/images/quick-start/Redis Connect Swagger Front Page.jpg" style="float: right;" width = 700px height = 425px/> |
| 129 | + |
| 130 | +**Create Job Configuration** - `/connect/api/vi/job/config/{jobName}` |
| 131 | +<br>_For quick start, use the sample `cdc-job.json` configuration:_ <a href="/examples/postgres/demo/config/samples/payloads/cdc-job.json">Gemfire</a> |
| 132 | +<br><br><img src="/images/quick-start/Redis Connect Save Job Config.png" style="float: right;" width = 700px height = 375px/> |
| 133 | +<br> |
| 134 | + |
| 135 | +**Or Use `curl` to create the `cdc-job` configuration** <br> |
| 136 | +`demo$ curl -v -X POST "http://localhost:8282/connect/api/v1/job/config/cdc-job" -H "accept: */*" -H "Content-Type: multipart/form-data" -F "file=@config/samples/payloads/cdc-job.json;type=application/json"` |
| 137 | + |
| 138 | +------------------------------- |
| 139 | + |
| 140 | +### Initial Loader Steps |
| 141 | + |
| 142 | +<details><summary><b>INSERT few records into postgres table (source)</b></summary> |
| 143 | +<p> |
| 144 | +You can also use <a href="https://github.com/redis-field-engineering/redis-connect-crud-loader#redis-connect-crud-loader">redis-connect-crud-loader</a> to insert load large amount of data using a csv or sql file. |
| 145 | + |
| 146 | +```bash |
| 147 | +demo$ ./load.sh |
| 148 | + |
| 149 | + |
| 150 | +``` |
| 151 | + |
| 152 | +</p> |
| 153 | +</details> |
| 154 | + |
| 155 | +**Start Job -** `/connect/api/vi/job/transition/start/{jobName}/{jobType}` |
| 156 | +<br>Use '**load**' as _**jobType**_ |
| 157 | +<br><br><img src="/images/quick-start/Redis Connect Start Job.png" style="float: right;" width = 700px height = 375px/> |
| 158 | + |
| 159 | +**Or Use `curl` to start the initial load for `cdc-job`** <br> |
| 160 | +`demo$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/cdc-job/load" -H "accept: */*"` |
| 161 | + |
| 162 | +<details><summary><b>Query for the above inserted record in Redis (target)</b></summary> |
| 163 | +<p> |
| 164 | + |
| 165 | +```bash |
| 166 | +demo$ |
| 167 | +``` |
| 168 | + |
| 169 | +</p> |
| 170 | +</details> |
| 171 | + |
| 172 | +------------------------------- |
| 173 | + |
| 174 | +### CDC Steps |
| 175 | + |
| 176 | +**Start Job -** `/connect/api/vi/job/transition/start/{jobName}/{jobType}` |
| 177 | +<br>Use '**stream**' as _**jobType**_ |
| 178 | +<br><br><img src="/images/quick-start/Redis Connect Start Job.png" style="float: right;" width = 700px height = 375px/> |
| 179 | + |
| 180 | +**Or Use `curl` to start the stream for `cdc-job`** <br> |
| 181 | +`demo$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/cdc-job/stream" -H "accept: */*"` |
| 182 | + |
| 183 | +**Confirm Job Claim -** `/connect/api/vi/jobs/claim/{jobStatus}` |
| 184 | +<br>_For quick start, use '**all**' as **jobStatus**_ |
| 185 | +<br><br><img src="/images/quick-start/Redis Connect Get Claims.png" style="float: right;" width = 700px height = 250px/> |
| 186 | + |
| 187 | +**Or Use `curl` to query the `cdc-job` status** <br> |
| 188 | +`demo$ curl -X GET "http://localhost:8282/connect/api/v1/cluster/jobs/claim/all" -H "accept: */*"` |
| 189 | + |
| 190 | +Expected output: `[{"jobId":"{connect}:job:cdc-job","jobName":"cdc-job","jobStatus":"CLAIMED","jobOwner":"30@virag-cdc","jobType":"STREAM"}]` |
| 191 | + |
| 192 | +<details><summary><b>INSERT a record into postgres table (source)</b></summary> |
| 193 | +<p> |
| 194 | + |
| 195 | +```bash |
| 196 | +demo$ ./insert.sh |
| 197 | + |
| 198 | + |
| 199 | +``` |
| 200 | + |
| 201 | +</p> |
| 202 | +</details> |
| 203 | + |
| 204 | +<details><summary><b>Query for the above inserted record in Redis (target)</b></summary> |
| 205 | +<p> |
| 206 | + |
| 207 | +```bash |
| 208 | +demo$ sudo docker exec -it re-node1 bash -c 'redis-cli -p 12000 idx_emp "@fname:allen"' |
| 209 | + |
| 210 | +``` |
| 211 | + |
| 212 | +</p> |
| 213 | +</details> |
0 commit comments