Configure Kafka Mirror Maker in Unsecured Environment

Introduction :

Kafka’s mirroring feature makes it possible to maintain a replica of an existing Kafka cluster. Kafka MirrorMaker is a tool to copy data from source cluster to destination cluster

 

How Kafka MirrorMaker Works ?

Mirroring works by using the MirrorMaker tool (part of Apache Kafka) to consume records from topics on the source cluster and then create a local copy on the destination cluster. MirrorMaker uses one (or more) consumers that read from the source cluster, and a producer that writes to the local (destination) cluster.

To run Kafka Mirror Maker configureĀ consumer.properties and producer.properties file as follows :

The following is an example of the consumer.properties file:

# vi consumer.properties
bootstrap.servers=<broker1.source.com>:<Port>,<broker2.source.com>:<Port>,<broker3.source.com>:<Port>
group.id=<Consumer-Group-id>

Note: Make Sure broker host and port are of source cluster

The following is an example of producer.properties file:

# vi producer.properties
bootstrap.servers=<broker1.destination.com>:<Port>,<broker2.destination.com>:<Port>,<broker3.destination.com>:<Port>
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer

Note: Make Sure broker host and port are of destination cluster

To run MirrorMaker, execute the following command:

# bin/kafka-run-class.sh kafka.tools.MirrorMaker 
--consumer.config consumer.properties --producer.config producer.properties 
--whitelist="<Topic-name>"

Leave a Reply

Your email address will not be published. Required fields are marked *