Kafka Broker creates new cluster id after docker restart if it can't find one

Programming
, ,
1

Issue

When restarting a Docker container with Kafka broker and Zookeeper, the Kafka broker creates a new cluster ID instead of using the old cluster ID stored in meta.properties. This causes an error to appear and the broker shuts down.

How can the broker be prevented from changing its cluster ID?

Steps to Reproduce:

  1. Start the Docker container with ./run_docker
  2. Shut down the Kafka broker and Zookeeper
  3. Start the Docker container again with ./run_docker
  4. The Kafka broker creates a new cluster ID instead of using the old one stored in meta.properties
  5. An error appears and the broker shuts down
ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentClusterIdException: The Cluster ID m1Ze6AjGRwqarkcxJscgyQ doesn't match stored clusterId Some(1TGYcbFuRXa4Lqojs4B9Hw) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.
        at kafka.server.KafkaServer.startup(KafkaServer.scala:220)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
        at kafka.Kafka$.main(Kafka.scala:84)
        at kafka.Kafka.main(Kafka.scala)
[2020-01-04 15:58:43,303] INFO shutting down (kafka.server.KafkaServer)
2

Add the following line to the Kafka broker configuration file (server.properties):

kafka.cluster.id=<cluster_id>

Replace <cluster_id> with the existing cluster ID stored in meta.properties. This will force the broker to use the same cluster ID instead of creating a new one.