Skip to content

Commit 8f8d2e3

Browse files
mimaisonGitHub Enterprise
authored andcommitted
Add sample and instructions for kafka-connect (#23)
* Add sample and instructions for kafka-connect * Add instructions to run connectors
1 parent 8b5a7f9 commit 8f8d2e3

File tree

8 files changed

+274
-0
lines changed

8 files changed

+274
-0
lines changed

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ In each sample, we demonstrate a single connection path for both our Classic and
2828
* [kafka-java-liberty-sample](/kafka-java-liberty-sample/README.md) : Sample IBM Websphere Liberty profile application using the Event Streams for IBM Cloud Kafka API
2929
* [kafka-nodejs-console-sample](kafka-nodejs-console-sample/README.md) : Sample Node.js console application using the Event Streams for IBM Cloud Kafka API
3030
* [kafka-python-console-sample](/kafka-python-console-sample/README.md) : Sample Python console application using the Event Streams for IBM Cloud Kafka API
31+
* [kafka-connect](/kafka-connect/README.md) : Sample Docker image with Kafka Connect
3132

3233
## Get Further Assistance
3334

kafka-connect/Dockerfile

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# This dockerfile expects Connector jars to have been built under a `connectors` directory
2+
#
3+
FROM ibmcom/eventstreams-kafka-ce-icp-linux-amd64:2019.2.1-3a2f93e as builder
4+
5+
FROM ibmjava:8-jre
6+
7+
RUN addgroup --gid 5000 --system esgroup && \
8+
adduser --uid 5000 --ingroup esgroup --system esuser
9+
10+
COPY --chown=esuser:esgroup --from=builder /opt/kafka/bin/ /opt/kafka/bin/
11+
COPY --chown=esuser:esgroup --from=builder /opt/kafka/libs/ /opt/kafka/libs/
12+
COPY --chown=esuser:esgroup --from=builder /opt/kafka/config/ /opt/kafka/config/
13+
RUN mkdir /opt/kafka/logs && chown esuser:esgroup /opt/kafka/logs
14+
15+
COPY --chown=esuser:esgroup connectors /opt/connectors
16+
17+
WORKDIR /opt/kafka
18+
19+
EXPOSE 8083
20+
21+
USER esuser
22+
23+
ENTRYPOINT ["./bin/connect-distributed.sh", "config/connect-distributed.properties"]

kafka-connect/IKS/README.md

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
2+
## Deploying `eventstreams-kafkaconnect` to Kubernetes connecting to IBM Event Streams
3+
4+
### Prerequisites
5+
6+
- `kubectl` access to a Kubernetes cluster.
7+
- Credentials to an IBM Event Streams instance that has the following permissions:
8+
- to create topics required by the Kafka Connect configuration (see `connect-distributed.properties`)
9+
- to read/write to the topics accessed by the Connectors
10+
11+
### Configure Kafka Connect
12+
13+
Edit `connect-distributed.properties` replacing the `<BOOTSTRAP_SERVERS>` and `<APIKEY>` placeholders with your Event Streams credentials.
14+
15+
Create the following Kubernetes resources:
16+
17+
```shell
18+
kubectl create secret generic connect-distributed-config --from-file=connect-distributed.properties
19+
kubectl create configmap connect-log4j-config --from-file=connect-log4j.properties
20+
```
21+
22+
### Run Kafka Connect in distributed mode in your Kubernetes cluster
23+
24+
Deploy the `ibmcom/eventstreams-kafkaconnect` Docker image:
25+
26+
```shell
27+
kubectl apply -f ./kafka-connect.yaml
28+
```
29+
Note that the sample yaml file specifies a single replica. Edit the `replicas` field if you want to run multiple Connect workers.
30+
Also, note that affinity rules might be needed to spread the workers across nodes.
31+
32+
### Manage Connectors
33+
34+
To manage connectors, port forward to the `kafkaconnect-service` Service on port 8083:
35+
36+
```shell
37+
kubectl port-forward service/kafkaconnect-service 8083
38+
```
39+
40+
The Connect REST API is then available via `http://localhost:8083`.
41+
The Connect REST API is documented at https://kafka.apache.org/documentation/#connect_rest
42+
43+
### Run Connectors
44+
45+
Once the Kafka Connect runtime is running, see the instructions for running the connectors:
46+
- [Run the COS Sink connector](https://github.com/ibm-messaging/kafka-connect-ibmcos-sink#running-the-connector)
47+
- [Run the MQ Source connector](https://github.com/ibm-messaging/kafka-connect-mq-source#running-the-connector)
Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
##
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
##
17+
18+
# This file contains some of the configurations for the Kafka Connect distributed worker. This file is intended
19+
# to be used with the examples, and some settings may differ from those used in a production system, especially
20+
# the `bootstrap.servers` and those specifying replication factors.
21+
22+
# A list of host/port pairs to use for establishing the initial connection to the Kafka cluster.
23+
bootstrap.servers=<BOOTSTRAP_SERVERS>
24+
security.protocol=SASL_SSL
25+
sasl.mechanism=PLAIN
26+
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="token" password="<APIKEY>";
27+
28+
consumer.security.protocol=SASL_SSL
29+
consumer.sasl.mechanism=PLAIN
30+
consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="token" password="<APIKEY>";
31+
32+
producer.security.protocol=SASL_SSL
33+
producer.sasl.mechanism=PLAIN
34+
producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="token" password="<APIKEY>";
35+
36+
plugin.path=/opt/connectors
37+
38+
# unique name for the cluster, used in forming the Connect cluster group. Note that this must not conflict with consumer group IDs
39+
group.id=connect-cluster
40+
41+
# The converters specify the format of data in Kafka and how to translate it into Connect data. Every Connect user will
42+
# need to configure these based on the format they want their data in when loaded from or stored into Kafka
43+
key.converter=org.apache.kafka.connect.json.JsonConverter
44+
value.converter=org.apache.kafka.connect.json.JsonConverter
45+
# Converter-specific settings can be passed in by prefixing the Converter's setting with the converter we want to apply
46+
# it to
47+
key.converter.schemas.enable=true
48+
value.converter.schemas.enable=true
49+
50+
# Topic to use for storing offsets. This topic should have many partitions and be replicated and compacted.
51+
# Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create
52+
# the topic before starting Kafka Connect if a specific topic configuration is needed.
53+
# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value.
54+
# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able
55+
# to run this example on a single-broker cluster and so here we instead set the replication factor to 1.
56+
offset.storage.topic=connect-offsets
57+
offset.storage.replication.factor=3
58+
#offset.storage.partitions=25
59+
60+
# Topic to use for storing connector and task configurations; note that this should be a single partition, highly replicated,
61+
# and compacted topic. Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create
62+
# the topic before starting Kafka Connect if a specific topic configuration is needed.
63+
# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value.
64+
# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able
65+
# to run this example on a single-broker cluster and so here we instead set the replication factor to 1.
66+
config.storage.topic=connect-configs
67+
config.storage.replication.factor=3
68+
69+
# Topic to use for storing statuses. This topic can have multiple partitions and should be replicated and compacted.
70+
# Kafka Connect will attempt to create the topic automatically when needed, but you can always manually create
71+
# the topic before starting Kafka Connect if a specific topic configuration is needed.
72+
# Most users will want to use the built-in default replication factor of 3 or in some cases even specify a larger value.
73+
# Since this means there must be at least as many brokers as the maximum replication factor used, we'd like to be able
74+
# to run this example on a single-broker cluster and so here we instead set the replication factor to 1.
75+
status.storage.topic=connect-status
76+
status.storage.replication.factor=3
77+
#status.storage.partitions=5
78+
79+
# Flush much faster than normal, which is useful for testing/debugging
80+
offset.flush.interval.ms=10000
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
log4j.rootLogger=INFO, stdout
2+
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
3+
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
4+
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c:%L)%n
5+
log4j.logger.org.apache.zookeeper=ERROR
6+
log4j.logger.org.I0Itec.zkclient=ERROR
7+
log4j.logger.org.reflections=ERROR
8+
# switch to TRACE for debugging the IBM connectors
9+
log4j.logger.com.ibm.eventstreams=INFO
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Deployment
2+
apiVersion: apps/v1
3+
kind: Deployment
4+
metadata:
5+
name: kafkaconnect-deploy
6+
labels:
7+
app: kafkaconnect
8+
spec:
9+
replicas: 1
10+
selector:
11+
matchLabels:
12+
app: kafkaconnect
13+
template:
14+
metadata:
15+
namespace: default
16+
labels:
17+
app: kafkaconnect
18+
spec:
19+
securityContext:
20+
runAsNonRoot: true
21+
runAsUser: 5000
22+
containers:
23+
- name: kafkaconnect-container
24+
image: ibmcom/eventstreams-kafkaconnect:latest
25+
readinessProbe:
26+
httpGet:
27+
path: /
28+
port: 8083
29+
livenessProbe:
30+
httpGet:
31+
path: /
32+
port: 8083
33+
ports:
34+
- containerPort: 8083
35+
volumeMounts:
36+
- name: connect-config
37+
mountPath: /opt/kafka/config/connect-distributed.properties
38+
subPath: connect-distributed.properties
39+
- name: connect-log4j
40+
mountPath: /opt/kafka/config/connect-log4j.properties
41+
subPath: connect-log4j.properties
42+
volumes:
43+
- name: connect-config
44+
secret:
45+
secretName: connect-distributed-config
46+
- name: connect-log4j
47+
configMap:
48+
name: connect-log4j-config
49+
---
50+
# Service
51+
apiVersion: v1
52+
kind: Service
53+
metadata:
54+
name: kafkaconnect-service
55+
labels:
56+
app: kafkaconnect
57+
spec:
58+
ports:
59+
- name: kafkaconnect
60+
protocol: TCP
61+
port: 8083
62+
selector:
63+
app: kafkaconnect

kafka-connect/README.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
# kafka-connect
2+
3+
This repository contains the artifacts required to build the `ibmcom/eventstreams-kafkaconnect` Docker image.
4+
5+
This image contains the Kafka Connect runtime and the [IBM Cloud Object Storage sink connector](https://github.com/ibm-messaging/kafka-connect-ibmcos-sink) and the [IBM MQ source connector](https://github.com/ibm-messaging/kafka-connect-mq-source).
6+
7+
A prebuilt image is provided on DockerHub: https://hub.docker.com/r/ibmcom/eventstreams-kafkaconnect.
8+
9+
## Running the image in Kubernetes
10+
11+
Instructions for running the `eventstreams-kafkaconnect` image in Kubernetes can be found [here](IKS/README.md).
12+
13+
## Building the image
14+
15+
In case you don't want to use the image we provide, you can build an image by completing these steps:
16+
17+
1. Run the `build.sh` script to download and compile the connectors:
18+
```shell
19+
./build.sh
20+
```
21+
22+
2. Build the docker image
23+
```shell
24+
docker build .
25+
```
26+
If you want to use the sample [YAML file](IKS/kafka-connect.yaml), be sure to update the image name with your image name.

kafka-connect/build.sh

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#!/bin/sh
2+
# Builds the Connectors for the Dockerfile requires Maven, git and gradle
3+
set -e
4+
5+
# Cleanup
6+
rm -rf ./connectors/*
7+
rm -rf ./kafka-connect-ibmcos-sink
8+
rm -rf ./kafka-connect-mq-source
9+
mkdir -p ./connectors
10+
11+
# Build COS Sink
12+
(
13+
git clone https://github.com/ibm-messaging/kafka-connect-ibmcos-sink
14+
cd kafka-connect-ibmcos-sink
15+
gradle clean shadowJar
16+
cp ./build/libs/kafka-connect-ibmcos-sink-*-all.jar ../connectors
17+
)
18+
19+
# Build MQ Source
20+
(
21+
git clone https://github.com/ibm-messaging/kafka-connect-mq-source
22+
cd kafka-connect-mq-source
23+
mvn clean package
24+
cp ./target/kafka-connect-mq-source-*-jar-with-dependencies.jar ../connectors
25+
)

0 commit comments

Comments
 (0)