Skip to content

Commit 23c3d11

Browse files
jeqojeqo
committed
docs: add Docker demo on how to install
1 parent 02e1d15 commit 23c3d11

File tree

5 files changed

+184
-1
lines changed

5 files changed

+184
-1
lines changed

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,8 @@ A collection of [Single Message Transformations (SMTs)](https://kafka.apache.org
44

55
## Transformations
66

7-
See [the Kafka documentation](https://kafka.apache.org/documentation/#connect_transforms) for more details about configuring transformations.
7+
See [the Kafka documentation](https://kafka.apache.org/documentation/#connect_transforms) for more details about configuring transformations
8+
or [demo](./demo) on how to install transforms.
89

910
### `ExtractTimestamp`
1011

‎demo/Dockerfile‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
FROM eclipse-temurin:11-jdk AS base
2+
3+
ADD ./ transforms
4+
WORKDIR transforms
5+
6+
RUN ./gradlew installDist
7+
8+
FROM confluentinc/cp-kafka-connect:7.3.3
9+
10+
COPY --from=base /transforms/build/install/transforms-for-apache-kafka-connect /transforms

‎demo/Makefile‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
all: build
2+
3+
build:
4+
docker compose build
5+
6+
up:
7+
docker compose up -d
8+
9+
test:
10+
curl -sSL 'http://localhost:8083/connector-plugins?connectorsOnly=false' \
11+
| jq '.[] | select(.type=="transformation") | select (.class | startswith("io.aiven"))'

‎demo/README.md‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,116 @@
1+
# Demo
2+
3+
How to load plugins to Kafka Connect installations.
4+
5+
## Requirements
6+
7+
To run the demo, you need:
8+
- Docker compose
9+
- `curl`
10+
- `jq`
11+
12+
## Demo
13+
14+
To install transforms into a Kafka Connect installation, it needs:
15+
16+
- Build `transforms-for-apache-kafka-connect` libraries
17+
- Add libraries to Kafka Connect nodes
18+
- Configure Kafka Connect `plugin.path`
19+
20+
### Build `transforms-for-apache-kafka-connect` libraries
21+
22+
To build libraries, use `gradlew installDist` command.
23+
e.g. Dockerfile build:
24+
25+
```dockerfile
26+
FROM eclipse-temurin:11-jdk AS base
27+
28+
ADD ./ transforms
29+
WORKDIR transforms
30+
31+
RUN ./gradlew installDist
32+
```
33+
34+
This generates the set of libraries to be installed in Kafka Connect workers.
35+
36+
### Add libraries to Kafka Connect nodes
37+
38+
Copy the directory with libraries into your Kafka Connect nodes.
39+
e.g. add directory to Docker images:
40+
41+
```dockerfile
42+
FROM confluentinc/cp-kafka-connect:7.3.3
43+
44+
COPY --from=base /transforms/build/install/transforms-for-apache-kafka-connect /transforms
45+
```
46+
47+
### Configure Kafka Connect `plugin.path`
48+
49+
On Kafka Connect configuration file, set `plugin.path` to indicate where to load plugins from,
50+
e.g. with Docker compose:
51+
52+
```yaml
53+
connect:
54+
# ...
55+
environment:
56+
# ...
57+
CONNECT_PLUGIN_PATH: /usr/share/java,/transforms # /transforms added on Dockerfile build
58+
```
59+
60+
## Running
61+
62+
1. Build docker images: `make build` or `docker compose build`
63+
2. Start environment: `make up` or `docker compose up -d`
64+
3. Test connect plugins are loaded: `make test`
65+
66+
Sample response:
67+
```json lines
68+
{
69+
"class": "io.aiven.kafka.connect.transforms.ConcatFields$Key",
70+
"type": "transformation"
71+
}
72+
{
73+
"class": "io.aiven.kafka.connect.transforms.ConcatFields$Value",
74+
"type": "transformation"
75+
}
76+
{
77+
"class": "io.aiven.kafka.connect.transforms.ExtractTimestamp$Key",
78+
"type": "transformation"
79+
}
80+
{
81+
"class": "io.aiven.kafka.connect.transforms.ExtractTimestamp$Value",
82+
"type": "transformation"
83+
}
84+
{
85+
"class": "io.aiven.kafka.connect.transforms.ExtractTopic$Key",
86+
"type": "transformation"
87+
}
88+
{
89+
"class": "io.aiven.kafka.connect.transforms.ExtractTopic$Value",
90+
"type": "transformation"
91+
}
92+
{
93+
"class": "io.aiven.kafka.connect.transforms.FilterByFieldValue$Key",
94+
"type": "transformation"
95+
}
96+
{
97+
"class": "io.aiven.kafka.connect.transforms.FilterByFieldValue$Value",
98+
"type": "transformation"
99+
}
100+
{
101+
"class": "io.aiven.kafka.connect.transforms.Hash$Key",
102+
"type": "transformation"
103+
}
104+
{
105+
"class": "io.aiven.kafka.connect.transforms.Hash$Value",
106+
"type": "transformation"
107+
}
108+
{
109+
"class": "io.aiven.kafka.connect.transforms.MakeTombstone",
110+
"type": "transformation"
111+
}
112+
{
113+
"class": "io.aiven.kafka.connect.transforms.TombstoneHandler",
114+
"type": "transformation"
115+
}
116+
```

‎demo/compose.yml‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
version: '3.8'
2+
services:
3+
zookeeper:
4+
image: "confluentinc/cp-zookeeper:7.3.3"
5+
ports:
6+
- "2181:2181"
7+
environment:
8+
ZOOKEEPER_CLIENT_PORT: 2181
9+
10+
kafka:
11+
image: "confluentinc/cp-kafka:7.3.3"
12+
ports:
13+
- "9092:9092"
14+
environment:
15+
KAFKA_BROKER_ID: 1
16+
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
17+
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
18+
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,PLAINTEXT_HOST://localhost:9092
19+
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
20+
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
21+
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
22+
23+
connect:
24+
build:
25+
context: ../.
26+
dockerfile: demo/Dockerfile
27+
ports:
28+
- "8083:8083"
29+
environment:
30+
CONNECT_BOOTSTRAP_SERVERS: kafka:19092
31+
CONNECT_REST_ADVERTISED_HOST_NAME: localhost
32+
CONNECT_REST_PORT: 8083
33+
CONNECT_GROUP_ID: connect-demo
34+
CONNECT_CONFIG_STORAGE_TOPIC: connect-demo-configs
35+
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
36+
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
37+
CONNECT_OFFSET_STORAGE_TOPIC: connect-demo-offsets
38+
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
39+
CONNECT_STATUS_STORAGE_TOPIC: connect-demo-status
40+
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
41+
42+
CONNECT_KEY_CONVERTER: org.apache.kafka.connect.storage.StringConverter
43+
CONNECT_KEY_CONVERTER_SCHEMA_ENABLE: "false"
44+
CONNECT_VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
45+
CONNECT_PLUGIN_PATH: /usr/share/java,/transforms # /transforms added on Dockerfile build

0 commit comments

Comments
 (0)