Skip to content

Commit b457379

Browse files
slorello89slorello89
authored
Merge pull request #14 from redis-field-engineering/gemfire-demo
Gemfire additional data types demo
2 parents 5cda2f4 + 64c38a8 commit b457379

18 files changed

+325
-23
lines changed

‎examples/gemfire/demo/README.md‎

Lines changed: 159 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22
:white_check_mark: Setup and start Gemfire database (Source)<br>
33
:white_check_mark: Setup and start Redis Enterprise database (Target)<br>
44
:white_check_mark: Setup and start Redis Connect<br>
5-
:white_check_mark: Perform Initial load and CDC with Redis Connect<br>
5+
:white_check_mark: Perform Initial load and CDC with Redis Connect on Strings<br>
6+
:white_check_mark: Perform Initial load with Redis Connect on complex data types<br>
67

78
# Prerequisites
89
Docker compatible [*nix OS](https://en.wikipedia.org/wiki/Unix-like) and [Docker](https://docs.docker.com/get-docker) installed.
@@ -14,10 +15,10 @@ i.e.</br>
1415
wget -c https://github.com/redis-field-engineering/redis-connect-dist/archive/main.zip && \
1516
mkdir -p redis-connect/demo && \
1617
unzip main.zip "redis-connect-dist-main/examples/gemfire/*" -d redis-connect && \
17-
cp -R redis-connect/redis-connect-dist-main/examples/gemfire/demo/* redis-connect/demo && \
18-
rm -rf main.zip redis-connect/redis-connect-dist-main && \
1918
cd redis-connect && \
19+
cp -R redis-connect-dist-main/examples/gemfire/demo/* ./demo && \
2020
chmod a+x demo/*.sh && \
21+
rm -rf ./redis-connect-dist-main && rm ../main.zip && \
2122
cd demo
2223
```
2324

@@ -365,3 +366,158 @@ demo$
365366

366367
</p>
367368
</details>
369+
370+
-------------------------------
371+
372+
## More Complex Data Types
373+
374+
This demo also includes an example of using more complex Java types within Gemfire. The `extlib` folder contains two jar files:
375+
1. `gemfire-pojo-1.0.jar`
376+
2. `redis-connect-custom-stage-sample-*.jar`
377+
378+
The first jar contains the code for a `redis.gemfire.Customer` class, which is a very simple POJO containing a name and age. The second jar is a build of [redis-connect-custom-stage](https://github.com/redis-field-engineering/redis-connect-custom-stage-sample) containing some custom staging code to transform arbitrary data types (in this case a `Customer`) into a type that redis-connect can serialize to `JSON`.
379+
380+
### Configure Job
381+
382+
On [http://localhost:8282/swagger-ui/index.html]() go to:
383+
384+
**Create Job Configuration** - `/connect/api/vi/job/config/{jobName}`
385+
<br>_For the customer POJO, use the sample `customer-pojo-job.json` configuration: <a href="/examples/postgres/demo/config/samples/payloads/cdc-job.json">Gemfire</a>
386+
<br>Use '**customer-pojo-job**' as the _**jobName**_
387+
<br><br><img src="/images/quick-start/Redis Connect Save Job Config.png" style="float: right;" width = 700px height = 375px/>
388+
<br>
389+
390+
Or use `curl` to create the `customer-pojo-job` configuration:
391+
392+
`$ curl -v -X POST "http://localhost:8282/connect/api/v1/job/config/customer-pojo-job" -H "accept: */*" -H "Content-Type: multipart/form-data" -F "file=@config/samples/payloads/customer-pojo-job.json;type=application/json"`
393+
394+
### Start Load Job
395+
396+
<details>
397+
<summary>INSERT customer records into gemfire</summary>
398+
<p>
399+
400+
```bash
401+
$ ./load_customers.sh
402+
Inserting records in session region..
403+
404+
(1) Executing - connect --locator localhost[10334]
405+
406+
Connecting to Locator at [host=localhost, port=10334] ..
407+
Connecting to Manager at [host=749a3ef94bf8, port=1099] ..
408+
Successfully connected to: [host=749a3ef94bf8, port=1099]
409+
410+
You are connected to a cluster of version: 1.15.1
411+
412+
413+
(2) Executing - put --key="customer1" --value=("name":"Jack","age":35) --region=customer --value-class=redis.gemfire.Customer
414+
415+
Result : true
416+
Key Class : java.lang.String
417+
Key : customer1
418+
Value Class : redis.gemfire.Customer
419+
Old Value : null
420+
421+
422+
(3) Executing - put --key="customer2" --value=("name":"Alice","age":36) --region=customer --value-class=redis.gemfire.Customer
423+
424+
Result : true
425+
Key Class : java.lang.String
426+
Key : customer2
427+
Value Class : redis.gemfire.Customer
428+
Old Value : null
429+
430+
431+
(4) Executing - put --key="customer3" --value=("name":"Bob","age":37) --region=customer --value-class=redis.gemfire.Customer
432+
433+
Result : true
434+
Key Class : java.lang.String
435+
Key : customer3
436+
Value Class : redis.gemfire.Customer
437+
Old Value : null
438+
439+
440+
(5) Executing - put --key="customer4" --value=("name":"Carol","age":38) --region=customer --value-class=redis.gemfire.Customer
441+
442+
Result : true
443+
Key Class : java.lang.String
444+
Key : customer4
445+
Value Class : redis.gemfire.Customer
446+
Old Value : null
447+
448+
449+
(6) Executing - put --key="customer5" --value=("name":"David","age":39) --region=customer --value-class=redis.gemfire.Customer
450+
451+
Result : true
452+
Key Class : java.lang.String
453+
Key : customer5
454+
Value Class : redis.gemfire.Customer
455+
Old Value : null
456+
457+
458+
(7) Executing - put --key="customer6" --value=("name":"Eva","age":40) --region=customer --value-class=redis.gemfire.Customer
459+
460+
Result : true
461+
Key Class : java.lang.String
462+
Key : customer6
463+
Value Class : redis.gemfire.Customer
464+
Old Value : null
465+
466+
467+
(8) Executing - put --key="customer7" --value=("name":"Frank","age":41) --region=customer --value-class=redis.gemfire.Customer
468+
469+
Result : true
470+
Key Class : java.lang.String
471+
Key : customer7
472+
Value Class : redis.gemfire.Customer
473+
Old Value : null
474+
475+
476+
(9) Executing - put --key="customer8" --value=("name":"Grace","age":42) --region=customer --value-class=redis.gemfire.Customer
477+
478+
Result : true
479+
Key Class : java.lang.String
480+
Key : customer8
481+
Value Class : redis.gemfire.Customer
482+
Old Value : null
483+
484+
485+
(10) Executing - put --key="customer9" --value=("name":"Henry","age":43) --region=customer --value-class=redis.gemfire.Customer
486+
487+
Result : true
488+
Key Class : java.lang.String
489+
Key : customer9
490+
Value Class : redis.gemfire.Customer
491+
Old Value : null
492+
493+
494+
(11) Executing - put --key="customer10" --value=("name":"Ivy","age":44) --region=customer --value-class=redis.gemfire.Customer
495+
496+
Result : true
497+
Key Class : java.lang.String
498+
Key : customer10
499+
Value Class : redis.gemfire.Customer
500+
Old Value : null
501+
502+
done
503+
```
504+
505+
</p>
506+
</details>
507+
508+
**Start Job -** `/connect/api/vi/job/transition/start/{jobName}/{jobType}`
509+
<br>Use '**load**' as _**jobType**_ and '**customer-pojo-job**' as the _**jobName**_
510+
<br><br><img src="/images/quick-start/Redis Connect Start Job.png" style="float: right;" width = 700px height = 375px/>
511+
512+
**Or Use `curl` to start the initial load for `customer-pojo-job`** <br>
513+
`$ curl -X POST "http://localhost:8282/connect/api/v1/job/transition/start/customer-pojo-job/load" -H "accept: */*"`
514+
515+
### Query Data in Redis
516+
517+
Your data is now available in Redis at the keys `{gemfireRegionName}:{gemfireKeyName}` and can be queried in Redis using the redis-cli:
518+
519+
520+
```bash
521+
$ redis-cli -p 14000 JSON.GET customer:customer1 $
522+
"[{\"name\":\"Jack\",\"age\":35}]"
523+
```

‎examples/gemfire/demo/cache.xml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,4 +30,16 @@
3030
<value-constraint>java.lang.String</value-constraint>
3131
</region-attributes>
3232
</region>
33+
<region name="customer">
34+
<region-attributes refid="PARTITION">
35+
<key-constraint>java.lang.String</key-constraint>
36+
<value-constraint>redis.gemfire.Customer</value-constraint>
37+
</region-attributes>
38+
</region>
39+
<region name="hash">
40+
<region-attributes refid="PARTITION">
41+
<key-constraint>java.lang.String</key-constraint>
42+
<value-constraint>java.util.HashMap</value-constraint>
43+
</region-attributes>
44+
</region>
3345
</cache>

‎examples/gemfire/demo/config/samples/credentials/redisconnect_credentials_gemfire_customer-pojo-job.properties‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
source.username=
2+
source.password=
3+
ssl-keystore-password=
4+
ssl-truststore-password=

‎examples/gemfire/demo/config/samples/credentials/redisconnect_credentials_gemfire_hash-job.properties‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
source.username=
2+
source.password=
3+
ssl-keystore-password=
4+
ssl-truststore-password=

‎examples/gemfire/demo/config/samples/credentials/redisconnect_credentials_redis_customer-pojo-job.properties‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
target.username=
2+
target.password=
3+
target.truststore.password=
4+
target.keystore.password=

‎examples/gemfire/demo/config/samples/credentials/redisconnect_credentials_redis_hash-job.properties‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
target.username=
2+
target.password=
3+
target.truststore.password=
4+
target.keystore.password=

‎examples/gemfire/demo/config/samples/payloads/cdc-job.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
"stages": [
66
{
77
"database": {
8-
"credentialsFilePath" : "/opt/redislabs/redis-connect/config/samples/credentials",
8+
"credentialsDirectoryPath" : "/opt/redislabs/redis-connect/config/samples/credentials",
99
"databaseURL": "redis://127.0.0.1:14000",
1010
"databaseType": "REDIS",
1111
"customConfiguration": {
@@ -22,7 +22,7 @@
2222
},
2323
"source": {
2424
"database": {
25-
"credentialsFilePath": "/opt/redislabs/redis-connect/config/samples/credentials",
25+
"credentialsDirectoryPath": "/opt/redislabs/redis-connect/config/samples/credentials",
2626
"databaseType": "GEMFIRE",
2727
"customConfiguration": {
2828
"pool.locator.host": "localhost",

‎examples/gemfire/demo/config/samples/payloads/customer-pojo-job.json‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
{
2+
"partitions": 1,
3+
"maxPartitionsPerClusterMember": 1,
4+
"pipeline": {
5+
"stages": [
6+
{
7+
"index": 1,
8+
"stageName": "OBJECT_TO_MAP_STAGE",
9+
"userDefinedType": "CUSTOM",
10+
"metricsEnabled": false
11+
},
12+
{
13+
"database": {
14+
"credentialsDirectoryPath" : "/opt/redislabs/redis-connect/config/samples/credentials",
15+
"databaseURL": "redis://127.0.0.1:14000",
16+
"databaseType": "REDIS",
17+
"customConfiguration": {
18+
"redis.connection.sslEnabled": false,
19+
"truststore": "/opt/redislabs/redis-connect/config/samples/credentials/client-truststore.jks"
20+
}
21+
},
22+
"index": 2,
23+
"stageName": "REDIS_JSON_SINK",
24+
"checkpointStageIndicator": true,
25+
"metricsEnabled" : true
26+
}
27+
]
28+
},
29+
"source": {
30+
"database": {
31+
"credentialsDirectoryPath": "/opt/redislabs/redis-connect/config/samples/credentials",
32+
"databaseType": "GEMFIRE",
33+
"customConfiguration": {
34+
"pool.locator.host": "localhost",
35+
"pool.locator.port": "10334"
36+
}
37+
},
38+
"tables": {
39+
"customer": {
40+
"initialLoad": {
41+
"partitions": 4,
42+
"maxPartitionsPerClusterMember": 1
43+
},
44+
"autoConfigColumnsEnabled": true
45+
}
46+
}
47+
}
48+
}

‎examples/gemfire/demo/config/samples/payloads/hash-job.json‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
{
2+
"partitions": 1,
3+
"maxPartitionsPerClusterMember": 1,
4+
"pipeline": {
5+
"stages": [
6+
{
7+
"index": 1,
8+
"stageName": "GEMFIRE_HASH_PREP_STAGE",
9+
"userDefinedType": "CUSTOM",
10+
"metricsEnabled": false
11+
},
12+
{
13+
"database": {
14+
"credentialsDirectoryPath" : "/opt/redislabs/redis-connect/config/samples/credentials",
15+
"databaseURL": "redis://127.0.0.1:14000",
16+
"databaseType": "REDIS",
17+
"customConfiguration": {
18+
"redis.connection.sslEnabled": false,
19+
"truststore": "/opt/redislabs/redis-connect/config/samples/credentials/client-truststore.jks"
20+
}
21+
},
22+
"index": 2,
23+
"stageName": "REDIS_HASH_SINK",
24+
"checkpointStageIndicator": true,
25+
"metricsEnabled" : true
26+
}
27+
]
28+
},
29+
"source": {
30+
"database": {
31+
"credentialsDirectoryPath": "/opt/redislabs/redis-connect/config/samples/credentials",
32+
"databaseType": "GEMFIRE",
33+
"customConfiguration": {
34+
"pool.locator.host": "localhost",
35+
"pool.locator.port": "10334"
36+
}
37+
},
38+
"tables": {
39+
"hash": {
40+
"initialLoad": {
41+
"partitions": 4,
42+
"maxPartitionsPerClusterMember": 1
43+
},
44+
"autoConfigColumnsEnabled": true
45+
}
46+
}
47+
}
48+
}

‎examples/gemfire/demo/delete.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
#!/bin/bash
22

3-
version="${1:-1.12.9}"
3+
version="${1:-1.15.9}"
44

55
container_name="gemfire-$version-$(hostname)"
66

0 commit comments

Comments
 (0)