Kafka Topic Deletion in Hung State

After running the Kafka topic deletion command the topic is stuck in ‘Marked for deletion’ state :

# ./kafka-topics --delete --topic <topic_name> --zookeeper <zk_string>)

NOTE: Kafka topic deletion is only supported 0.9.0+ .

EXPLANATION: When topic deletion is invoked through the kafka-topics.sh script, it will create a znode as /admin/delete_topics/<topic_name>. This will fire a watch to invoke the deletion thread on this topic name. The ‘Marked for deletion’ status in the topic list command is based on the existence of the znode.

Once the deletion thread is triggered, the necessary actions will be performed. However, the topic deletion can fail if any of the following criteria is met:

a. broker hosting one of the replicas for that topic goes down
b. partition reassignment for partitions of that topic is in progress
c. preferred replica election for partitions of that topic is in progress (though this is not strictly required since it holds the controller lock for the entire duration from start to end)
d. ‘delete.topic.enable’ property set to false

Topic deletion should resume when:
a. broker hosting one of the replicas for that topic is started
b. preferred replica election for partitions of that topic completes
c. partition reassignment for partitions of that topic completes
d. ‘delete.topic.enable’ property set to true and applied to cluster

In some scenarios, the topic deletion process may abort and fail to recover. In these scenarios, the topic becomes stuck in ‘Marked for deletion’ state. For these situations, the last resort of using manual topic deletion is needed.

PROCEDURE:

1. First and foremost, check that the property ‘delete.topic.enable’ is set to true.

2. Check controller logs (delete-topics-thread) which will provide more insights as to why topic is not deleted.

3. Make sure that all brokers hosting partitions for that topic are up. Also, make sure there is no partition reassignment operation happening during this time. To check if a manual partition reassignment has been invoked and is running, check the following znode in zookeeper shell:

/admin/reassign_partitions

If all of the above have been checked and the deletion is still stuck, a last resort manual deletion of the topic is required. The following is a list of steps to perform manual topic deletion for topic name ‘TestTopic’:

1. Shut down all brokers

2. Cleanup physical data
On each broker host, move out all folders under the Kafka log.dir associated with the topic.

For example, for kafka.log.dir=/grid1,/grid2, move out all of the following directories:

/grid1/TestTopic-<X>
/grid2/TestTopic-<X>
where <X> is any number from 0 to (total partition count - 1).
3. Cleanup zookeeper data
Login to zookeeper:
# bin/zkCli.sh -server <zk_fqdn>:<zk_port>

Execute the following commands:
NOTE: Following commands assume ‘/’ as kafka zookeeper root. If different, please prepend.

zkcli> ls /brokers/topics
zkcli> rmr /brokers/topics/thistopic

zkcli> ls /admin/delete_topics
zkcli> rmr /admin/delete_topics/thistopic

zkcli> ls /config/topics
zkcli> rmr /config/topics/thistopic

4. Restart brokers

Leave a Reply

Your email address will not be published. Required fields are marked *