Skip to content

Commit 19b403b

Browse files
mimaisonGitHub Enterprise
mimaison
authored and
GitHub Enterprise
committed
Merge pull request #28 from messagehub/mirrormaker
Add Mirrormaker sample
2 parents 56ae606 + deca4bb commit 19b403b

File tree

9 files changed

+158
-0
lines changed

9 files changed

+158
-0
lines changed

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ In each sample, we demonstrate a single connection path for both our Classic and
2929
* [kafka-nodejs-console-sample](kafka-nodejs-console-sample/README.md) : Sample Node.js console application using the Event Streams for IBM Cloud Kafka API
3030
* [kafka-python-console-sample](/kafka-python-console-sample/README.md) : Sample Python console application using the Event Streams for IBM Cloud Kafka API
3131
* [kafka-connect](/kafka-connect/README.md) : Sample Docker image with Kafka Connect
32+
* [kafka-mirrormaker](/kafka-mirrormaker/README.md) : Sample Docker image with Kafka Mirror Maker
3233

3334
## Get Further Assistance
3435

‎kafka-mirrormaker/Dockerfile‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
FROM ibmcom/eventstreams-kafka-ce-icp-linux-amd64:2019.2.1-3a2f93e as builder
2+
3+
FROM ibmjava:8-jre
4+
5+
RUN addgroup --gid 5000 --system esgroup && \
6+
adduser --uid 5000 --ingroup esgroup --system esuser
7+
8+
COPY --chown=esuser:esgroup --from=builder /opt/kafka/bin/ /opt/kafka/bin/
9+
COPY --chown=esuser:esgroup --from=builder /opt/kafka/libs/ /opt/kafka/libs/
10+
COPY --chown=esuser:esgroup --from=builder /opt/kafka/config/ /opt/kafka/config/
11+
RUN mkdir /opt/kafka/logs && chown esuser:esgroup /opt/kafka/logs
12+
13+
COPY --chown=esuser:esgroup entrypoint.sh /opt/kafka
14+
15+
WORKDIR /opt/kafka
16+
17+
ENV TOPIC_REGEX=.*
18+
19+
USER esuser
20+
21+
ENTRYPOINT ["./entrypoint.sh"]

‎kafka-mirrormaker/IKS/README.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
2+
## Deploying `eventstreams-kafkamirrormaker` to Kubernetes to replicate data between 2 Event Streams clusters
3+
4+
These steps detail how to replicate data from a Kafka cluster (source) to another Kafka cluster (destination) using the `eventstreams-kafkamirrormaker` image.
5+
6+
### Prerequisites
7+
8+
- `kubectl` access to a Kubernetes cluster.
9+
- Credentials for an IBM Event Streams instance that has the following permissions:
10+
- to read/write to the topics
11+
12+
Mirror Maker does not automatically create topics in the destination cluster. You must create these topics before starting Mirror Maker.
13+
14+
### Configure Mirror Maker
15+
16+
Edit `source.properties` replacing the `<BOOTSTRAP_SERVERS>` and `<APIKEY>` placeholders with your Event Streams credentials for the source cluster.
17+
18+
Edit `destination.properties` replacing the `<BOOTSTRAP_SERVERS>` and `<APIKEY>` placeholders with your Event Streams credentials for the destination cluster.
19+
20+
Create the following Kubernetes resources:
21+
22+
```shell
23+
kubectl create secret generic source-config --from-file=source.properties
24+
kubectl create secret generic destination-config --from-file=destination.properties
25+
kubectl create configmap tools-log4j-config --from-file=tools-log4j.properties
26+
```
27+
28+
### Run Mirror Maker in your Kubernetes cluster
29+
30+
Deploy the `ibmcom/eventstreams-kafkamirrormaker` Docker image:
31+
32+
```shell
33+
kubectl apply -f ./kafka-mirrormaker.yaml
34+
```

‎kafka-mirrormaker/IKS/destination.properties‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
bootstrap.servers=<BOOTSTRAP_SERVERS>
2+
security.protocol=SASL_SSL
3+
sasl.mechanism=PLAIN
4+
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="token" password="<APIKEY>";
5+
ssl.protocol=TLSv1.2
6+
ssl.enabled.protocols=TLSv1.2
7+
ssl.endpoint.identification.algorithm=HTTPS
8+
9+
acks=all
10+
client.id=mirror_maker_producer

‎kafka-mirrormaker/IKS/kafka-mirrormaker.yaml‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
# Deployment
2+
apiVersion: apps/v1
3+
kind: Deployment
4+
metadata:
5+
name: kafkamirrormaker-deploy
6+
labels:
7+
app: kafkamirrormaker
8+
spec:
9+
replicas: 1
10+
selector:
11+
matchLabels:
12+
app: kafkamirrormaker
13+
template:
14+
metadata:
15+
namespace: default
16+
labels:
17+
app: kafkamirrormaker
18+
spec:
19+
securityContext:
20+
runAsNonRoot: true
21+
runAsUser: 5000
22+
containers:
23+
- name: kafkamirrormaker-container
24+
image: ibmcom/eventstreams-kafkamirrormaker:latest
25+
env:
26+
- name: TOPIC_REGEX
27+
value: "mytopic"
28+
volumeMounts:
29+
- name: source-config
30+
mountPath: /opt/kafka/config/source.properties
31+
subPath: source.properties
32+
- name: destination-config
33+
mountPath: /opt/kafka/config/destination.properties
34+
subPath: destination.properties
35+
- name: tools-log4j
36+
mountPath: /opt/kafka/config/tools-log4j.properties
37+
subPath: tools-log4j.properties
38+
volumes:
39+
- name: source-config
40+
secret:
41+
secretName: source-config
42+
- name: destination-config
43+
secret:
44+
secretName: destination-config
45+
- name: tools-log4j
46+
configMap:
47+
name: tools-log4j-config

‎kafka-mirrormaker/IKS/source.properties‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
bootstrap.servers=<BOOTSTRAP_SERVERS>
2+
security.protocol=SASL_SSL
3+
sasl.mechanism=PLAIN
4+
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="token" password="<APIKEY>";
5+
ssl.protocol=TLSv1.2
6+
ssl.enabled.protocols=TLSv1.2
7+
ssl.endpoint.identification.algorithm=HTTPS
8+
9+
exclude.internal.topics=true
10+
client.id=mirror_maker_consumer
11+
group.id=mirror_maker_consumer
12+
partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor

‎kafka-mirrormaker/IKS/tools-log4j.properties‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
log4j.rootLogger=INFO, stderr
2+
3+
log4j.appender.stderr=org.apache.log4j.ConsoleAppender
4+
log4j.appender.stderr.layout=org.apache.log4j.PatternLayout
5+
log4j.appender.stderr.layout.ConversionPattern=[%d] %p %m (%c)%n
6+
log4j.appender.stderr.Target=System.err

‎kafka-mirrormaker/README.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# kafka-mirrormaker
2+
3+
This repository contains the artifacts required to build the `ibmcom/eventstreams-kafkamirrormaker` Docker image.
4+
5+
This image contains [Kafka Mirror Maker](http://kafka.apache.org/documentation/#basic_ops_mirror_maker) and can be used to replicate data between clusters.
6+
7+
A prebuilt image is provided on DockerHub: https://hub.docker.com/r/ibmcom/eventstreams-kafkamirrormaker.
8+
9+
## Running the image in Kubernetes
10+
11+
Instructions for running the `eventstreams-kafkamirrormaker` image in Kubernetes can be found [here](IKS/README.md).
12+
13+
## Building the image
14+
15+
To build the image yourself, complete these steps:
16+
17+
1. Build the docker image:
18+
```shell
19+
docker build .
20+
```
21+
If you want to use the sample [YAML file](IKS/kafka-mirrormaker.yaml), ensure that you update the image name with your own image name.

‎kafka-mirrormaker/entrypoint.sh‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/bin/bash
2+
3+
./bin/kafka-mirror-maker.sh \
4+
--consumer.config config/source.properties \
5+
--producer.config config/destination.properties \
6+
--whitelist=${TOPIC_REGEX}

0 commit comments

Comments
 (0)