Skip to content

Commit 829a1c1

Browse files
viragtripathiviragtripathi
committed
Added example for Files connector.
1 parent 1721066 commit 829a1c1

17 files changed

+4103
-5
lines changed

‎README.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,8 @@ changed-data events from heterogeneous data platforms to [Redis Stack](https://r
2020
<tr>
2121
<td><a href="/examples/mongodb/demo/README.md"><img src="/images/icons/MongoDB Icon.png" style="float: center;" width="100" height="100" alt="Redis Connect MongoDB demo"/></a></td>
2222
<td><a href="/examples/gemfire/demo/README.md"><img src="/images/icons/Geode Icon.png" style="float: center;" width="100" height="100" alt="Redis Connect Gemfire/Apache Geode demo"/></a></td>
23-
<td><img src="/images/icons/Splunk HEC Icon.png" style="float: center;" width="100" height="100" alt="Redis Connect Splunk HEC demo"/></td>
24-
<td><img src="/images/icons/CSV Icon.png" style="float: center;" width="100" height="100" alt="Redis Connect Files demo"/></td>
23+
<td><a href="/examples/splunk/demo/README.md"><img src="/images/icons/Splunk HEC Icon.png" style="float: center;" width="100" height="100" alt="Redis Connect Splunk HEC demo"/></a>></td>
24+
<td><a href="/examples/files/demo/README.md"><img src="/images/icons/CSV Icon.png" style="float: center;" width="100" height="100" alt="Redis Connect Files demo"/></a>></td>
2525
<td><a href="/examples/vertica/demo/README.md"><img src="/images/icons/Vertica Icon.png" style="float: center;" width="100" height="100" alt="Redis Connect Vertica demo"/></a></td>
2626
</tr>
2727
</table>
@@ -158,7 +158,7 @@ The Redis Connect Swagger UI is available on port 8282 by default. If you're run
158158
<br><br><img src="/images/quick-start/Redis Connect Swagger Front Page.jpg" style="float: right;" width = 700px height = 425px/>
159159

160160
**Create Job Configuration** - `/connect/api/vi/job/config/{jobName}`
161-
<br>_For quick start, use a sample job configuration:_ <a href="/examples/db2/demo/config/samples/payloads/cdc-job.json">DB2</a>, <a href="/examples/gemfire/demo/config/samples/payloads/cdc-job.json">Gemfire</a>, <a href="/examples/mongodb/demo/config/samples/payloads/cdc-job.json">MongoDB</a>, <a href="/examples/mysql/demo/config/samples/payloads/cdc-job.json">MySQL</a>, <a href="/examples/oracle/demo/config/samples/payloads/cdc-job.json">Oracle</a>, <a href="/examples/postgres/demo/config/samples/payloads/cdc-job.json">PostgreSQL</a>, <a href="/examples/mssql/demo/config/samples/payloads/cdc-job.json">SQL Server</a> and <a href="/examples/vertica/demo/config/samples/payloads/cdc-job.json">VERTICA</a>
161+
<br>_For quick start, use a sample job configuration:_ <a href="/examples/db2/demo/config/samples/payloads/cdc-job.json">DB2</a>, <a href="/examples/files/demo/config/samples/payloads/cdc-job.json">Files</a>, <a href="/examples/gemfire/demo/config/samples/payloads/cdc-job.json">Gemfire</a>, <a href="/examples/mongodb/demo/config/samples/payloads/cdc-job.json">MongoDB</a>, <a href="/examples/mysql/demo/config/samples/payloads/cdc-job.json">MySQL</a>, <a href="/examples/oracle/demo/config/samples/payloads/cdc-job.json">Oracle</a>, <a href="/examples/postgres/demo/config/samples/payloads/cdc-job.json">PostgreSQL</a>, <a href="/examples/mssql/demo/config/samples/payloads/cdc-job.json">SQL Server</a> and <a href="/examples/vertica/demo/config/samples/payloads/cdc-job.json">VERTICA</a>
162162
<br><br><img src="/images/quick-start/Redis Connect Save Job Config.png" style="float: right;" width = 700px height = 375px/>
163163

164164
| Prerequisite Configuration :exclamation: |

‎examples/files/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
## Setting up Redis Enterprise Databases (Target)
2+
3+
Before using Redis Connect to capture the changes committed on Gemfire into Redis Enterprise Databases, first create a database for the metadata management and metrics provided by Redis Connect by creating a database with [RedisTimeSeries](https://redis.com/modules/redis-timeseries/) module enabled, see [Create Redis Enterprise Database](https://docs.redis.com/latest/rs/administering/creating-databases/#creating-a-new-redis-database) for reference. Then, create (or use an existing) another Redis Enterprise database (Target) to store the changes coming from PostgreSQL. Additionally, you can enable [RediSearch 2.0](https://redis.com/blog/introducing-redisearch-2-0/) module on the target database to enable secondary index with full-text search capabilities on the existing hashes where PostgreSQL changed events are being written at then [create an index, and start querying](https://oss.redis.com/redisearch/Commands/) the document in hashes.
4+
5+
| ℹ️ |
6+
|:-------------------------------------------------|
7+
| Docker demo: Follow the [Docker demo](demo) |
8+
| K8s Setup: Follow the [k8s-docs](../../k8s-docs) |

‎examples/files/demo/README.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
# Demo Outline
2+
:white_check_mark: Have a supported File type ready (Source)<br>
3+
:white_check_mark: Setup and start Redis Enterprise database (Target)<br>
4+
:white_check_mark: Setup and start Redis Connect<br>
5+
:white_check_mark: Perform Initial load and CDC with Redis Connect<br>
6+
7+
# Prerequisites
8+
Docker compatible [*nix OS](https://en.wikipedia.org/wiki/Unix-like) and [Docker](https://docs.docker.com/get-docker) installed.
9+
<br>Please have 8 vCPU*, 8GB RAM and 50GB storage for this demo to function properly. Adjust the resources based on your requirements. For HA, at least have 2 Redis Connect instances deployed on separate hosts.</br>
10+
<br>Execute the following commands (copy & paste) to download and setup Redis Connect and demo scripts.
11+
i.e.</br>
12+
13+
```bash
14+
wget -c https://github.com/redis-field-engineering/redis-connect-dist/archive/main.zip && \
15+
mkdir -p redis-connect/demo && \
16+
unzip main.zip "redis-connect-dist-main/examples/files/*" -d redis-connect && \
17+
cp -R redis-connect/redis-connect-dist-main/examples/files/demo/* redis-connect/demo && \
18+
rm -rf main.zip redis-connect/redis-connect-dist-main && \
19+
cd redis-connect && \
20+
chmod a+x demo/*.sh && \
21+
cd demo
22+
```
23+
24+
## Setup Redis Enterprise cluster, databases and RedisInsight in docker (Target)
25+
<br>Execute [setup_re.sh](setup_re.sh)</br>
26+
```bash
27+
demo$ ./setup_re.sh
28+
```
29+
**NOTE**
30+
31+
The above script will create a 1-node Redis Enterprise cluster in a docker container, [Create a target database with RediSearch module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a job management and metrics database with RedisTimeSeries module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a RediSearch index for emp Hash](https://redislabs.com/blog/getting-started-with-redisearch-2-0/), [Start a docker instance of grafana with Redis Data Source](https://redisgrafana.github.io/) and [Start an instance of RedisInsight](https://docs.redislabs.com/latest/ri/installing/install-docker/).
32+
33+
## Start Redis Connect
34+
35+
<details><summary>Review options by running Redis Connect docker container </summary>
36+
<p>
37+
38+
```bash
39+
docker run \
40+
-it --rm --privileged=true \
41+
--name redis-connect-$(hostname) \
42+
-v $(pwd)/config:/opt/redislabs/redis-connect/config \
43+
-v $(pwd)/config/samples/credentials:/opt/redislabs/redis-connect/config/samples/credentials \
44+
--net host \
45+
redislabs/redis-connect
46+
```
47+
48+
</p>
49+
</details>
50+
51+
<details><summary>Expected output:</summary>
52+
<p>
53+
54+
```bash
55+
-------------------------------
56+
Redis Connect startup script.
57+
*******************************
58+
Please ensure that these environment variables are correctly mapped before executing start and cli options. They can also be found in /opt/redislabs/redis-connect/bin/redisconnect.conf
59+
Example environment variables and volume mapping for docker based deployments
60+
-e REDISCONNECT_JOB_MANAGER_CONFIG_PATH=/opt/redislabs/redis-connect/config/jobmanager.properties [OPTIONAL]
61+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect/config/logback.xml [OPTIONAL]
62+
-e REDISCONNECT_JAVA_OPTIONS=-Xms1g -Xmx2g [OPTIONAL]
63+
-e REDISCONNECT_EXTLIB_DIR=/opt/redislabs/redis-connect/extlib [OPTIONAL]
64+
-v <HOST_PATH_TO_JOB_MANAGER_PROPERTIES>:/opt/redislabs/redis-connect/config
65+
-v <HOST_PATH_TO_CREDENTIALS>:/opt/redislabs/redis-connect/config/samples/credentials
66+
-v <HOST_PATH_TO_EXTLIB>:/opt/redislabs/redis-connect/extlib [OPTIONAL]
67+
-p 8282:8282
68+
69+
Usage: [-h|cli|start]
70+
options:
71+
-h: Print this help message and exit.
72+
-v: Print version.
73+
cli: init Redis Connect CLI
74+
start: init Redis Connect Instance (Cluster Member)
75+
-------------------------------
76+
```
77+
78+
</p>
79+
</details>
80+
81+
<details><summary><b>Start Redis Connect Instance</b></summary>
82+
<p>
83+
84+
```bash
85+
docker run \
86+
-it --rm --privileged=true \
87+
--name redis-connect-$(hostname) \
88+
-v $(pwd)/config:/opt/redislabs/redis-connect/config \
89+
-v $(pwd)/extlib:/opt/redislabs/redis-connect/extlib \
90+
--net host \
91+
redislabs/redis-connect start
92+
```
93+
94+
</p>
95+
</details>
96+
97+
<details><summary>Expected output:</summary>
98+
<p>
99+
100+
```bash
101+
-------------------------------
102+
Starting redis-connect v0.10.1.5 Instance using JAVA 11.0.19 on docker-desktop started by root in /opt/redislabs/redis-connect/bin
103+
Loading redis-connect Instance configuration from /opt/redislabs/redis-connect/config/jobmanager.properties
104+
Instance classpath /opt/redislabs/redis-connect/lib/*:/opt/redislabs/redis-connect/extlib/*
105+
Check redis-connect-manager-<PID>.log for cluster-level information, redis-connect-heartbeat-<PID>.log for heartbeat-lease renewals, and redis-connect-<PID>.log for the job-level information
106+
07:11:09.184 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
107+
/####### /## /## /###### /##
108+
| ##__ ## | ## |__/ /##__ ## | ##
109+
| ## \ ## /###### /####### /## /####### | ## \__/ /###### /####### /####### /###### /####### /######
110+
| #######/ /##__ ## /##__ ##| ## /##_____/ | ## /##__ ##| ##__ ##| ##__ ## /##__ ## /##_____/|_ ##_/
111+
| ##__ ##| ########| ## | ##| ##| ###### | ## | ## \ ##| ## \ ##| ## \ ##| ########| ## | ##
112+
| ## \ ##| ##_____/| ## | ##| ## \____ ## | ## ##| ## | ##| ## | ##| ## | ##| ##_____/| ## | ## /##
113+
| ## | ##| #######| #######| ## /#######/ | ######/| ######/| ## | ##| ## | ##| #######| ####### | ####/
114+
|__/ |__/ \_______/ \_______/|__/|_______/ \______/ \______/ |__/ |__/|__/ |__/ \_______/ \_______/ \___/ v0.10.1
115+
Powered by Redis Enterprise
116+
07:11:14.190 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
117+
07:11:14.930 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: JobManager
118+
07:11:14.934 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: JobReaper
119+
07:11:14.939 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: JobClaimer
120+
07:11:14.943 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: JobOrchestrator
121+
07:11:14.947 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: HeartbeatManager
122+
07:11:14.949 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: MetricsReporter
123+
07:11:14.951 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: CredentialsRotationEventListener
124+
07:11:14.953 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully established Redis connection with ClientId: JobManager ConnectionId: ChangeEventQueue
125+
07:11:14.986 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully created Job Claim Assignment Stream and Consumer Group
126+
07:11:14.988 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully started JobManager service
127+
07:11:14.988 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully started JobReaper service
128+
07:11:14.989 [main] INFO redis-connect-manager - Instance: 30@docker-desktop successfully started JobClaimer service
129+
07:11:14.990 [main] INFO redis-connect-manager - Instance: 30@docker-desktop Metrics are not enabled so MetricsReporter threadpool will not be instantiated
130+
07:11:16.630 [main] INFO redis-connect-manager - Instance: 30@docker-desktop started Redis Connect REST API listening on ["http-nio-8282"]
131+
07:11:16.630 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
132+
07:11:16.630 [main] INFO redis-connect-manager -
133+
07:11:16.630 [main] INFO redis-connect-manager - Started Redis Connect Instance v0.10.1
134+
07:11:16.630 [main] INFO redis-connect-manager -
135+
07:11:16.630 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
136+
07:11:24.997 [JOB_MANAGER_THREADPOOL-2] INFO redis-connect-manager - Instance: 30@docker-desktop was successfully elected Redis Connect cluster leader
137+
```
138+
139+
</p>
140+
</details>
141+
142+
**Open browser to access Swagger UI -** [http://localhost:8282/swagger-ui/index.html]()
143+
<br>_For quick start, use '**cdc_job**' as **jobName**_
144+
<br><br><img src="/images/quick-start/Redis Connect Swagger Front Page.jpg" style="float: right;" width = 700px height = 425px/>
145+
146+
**Create Job Configuration** - `/connect/api/vi/job/config/{jobName}`
147+
<br>_For quick start, use the sample `cdc-job.json` configuration:_ <a href="/examples/files/demo/config/samples/payloads/cdc-job.json">Files</a>
148+
<br><br><img src="/images/quick-start/Redis Connect Save Job Config.png" style="float: right;" width = 700px height = 375px/>
149+
<br>
150+
151+
**Or Use `curl` to create the `cdc-job` configuration** <br>
152+
`demo$ curl -v -X POST "http://localhost:8282/connect/api/v1/job/config/cdc-job" -H "accept: */*" -H "Content-Type: multipart/form-data" -F "file=@config/samples/payloads/cdc-job.json;type=application/json"`
153+
154+
-------------------------------
155+
156+
### Initial Loader Steps
157+
158+
**Start Job -** `/connect/api/vi/job/transition/start/{jobName}/{jobType}`
159+
<br>Use '**load**' as _**jobType**_
160+
<br><br><img src="/images/quick-start/Redis Connect Start Job.png" style="float: right;" width = 700px height = 375px/>
161+
162+
**Or Use `curl` to start the initial load for `cdc-job`** <br>
163+
`demo$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/cdc-job/load" -H "accept: */*"`
164+
165+
<details><summary><b>Query for the above inserted record in Redis (target)</b></summary>
166+
<p>
167+
168+
```bash
169+
demo$
170+
```
171+
172+
</p>
173+
</details>

‎examples/files/demo/cleanup_re.sh‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
#!/bin/bash
2+
3+
version="${1:-latest}"
4+
cleanup="${2:-yes}"
5+
6+
container_name="re-node1-$version-$(hostname)"
7+
8+
# delete the existing container if it exist
9+
if [ "${cleanup}" = "yes" ]; then
10+
echo "Stopping and removing ${container_name} docker container from $(hostname)."
11+
docker container stop "${container_name}"; docker container rm "${container_name}";
12+
else
13+
echo "Skipping removing ${container_name} docker container from $(hostname)."
14+
fi
15+
16+
echo "done"

‎examples/files/demo/config/jobmanager.properties‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
##################################################################################################
2+
##### Properties to start a Redis Connect cluster. Default values are commented out. #####
3+
##################################################################################################
4+
5+
##### Cluster properties
6+
#cluster.name=default
7+
#cluster.leader.heartbeat.lease.renewal.interval=5000
8+
#cluster.election.attempt.interval=5000
9+
#cluster.timeseries.metrics.enabled=false
10+
11+
##### Job Manager Services properties
12+
#job.manager.services.enabled=true
13+
#job.manager.services.threadpool.size=2
14+
#job.reap.attempt.interval=7000
15+
#job.claim.service.enabled=true
16+
#job.claim.attempt.interval=5000
17+
#job.claim.batch.size.per.attempt=4
18+
#job.claim.max.capacity=4
19+
#job.claim.heartbeat.lease.renewal.interval=10000
20+
21+
##### REST properties
22+
#rest.api.enabled=true
23+
#rest.api.port=8282
24+
25+
##### Job Manager Database properties
26+
redis.connection.url=redis://127.0.0.1:14001
27+
#redis.connection.sslEnabled=false
28+
#truststore=<path_to_JKS>
29+
#keystore=<path_to_PKCS>
30+
#redis.connection.insecure=false
31+
#redis.connection.timeout.duration=5
32+
#redis.connection.auto.reconnect=true
33+
#redis.connection.suspend.reconnect.on.protocol.failure=true
34+
35+
##### Credentials properties
36+
credentials.file.path=/opt/redislabs/redis-connect/config/samples/credentials
37+
#credentials.rotation.event.listener.enabled=false
38+
#credentials.rotation.event.listener.interval=60000
39+
40+
##### Email Alert properties
41+
#mail.alert.enabled=false
42+
#mail.smtp.host=smtp.gmail.com
43+
#mail.smtp.port=587
44+
#mail.smtp.start.tls.enable=true
45+
#mail.smtp.start.tls.required=false
46+
#mail.to=
47+
#mail.debug=false

0 commit comments

Comments
 (0)