Recently I came across a situation where I wanted to decide the heap size of zookeeper. Along with this zookeeper was failing with Out Of Memory issue.
At zookeeper startup, the process will read the last snapshot file and create a DataTree. DataTree has a hash table of znodes. The process deserialize the snapshot file (which has all znodes) and add each znodes present into the hash table. There are ephemeral znodes which would have been created in previous session, that also will be present in the snapshot and will be added into the hash table. At the end of adding all data nodes, the previous session ephemeral nodes which is present are deleted from Hash table. So if there are lot of ephemeral znodes from previous session, then zookeeper startup will need more heap to store all of them and they will be removed immediately.
To sum this up zookeeper stores znodes and latest snapshot in heap.
I did some research on ideal value/formula to set right heap size for zookeeper server. There is no such ideal value but we can calculate the value of heap size based on below factors :
1. The number of znodes in zookeeper
2. Number of components using zookeeper (for ex : Hbase, Kafka, Solr etc…)
3. Size of all the znodes in zookeeper
4. Size of snapshot in zookeeper
Below command will give the number of znodes (zk_znode_count) and its size(zk_approximate_data_size) in bytes.
# echo mntr | nc localhost 2181 zk_version 3.4.6-76--1, built on 11/07/2017 01:38 GMT zk_avg_latency 0 zk_max_latency 1422 zk_min_latency 0 zk_packets_received 31318142 zk_packets_sent 31318171 zk_num_alive_connections 3 zk_outstanding_requests 0 zk_server_state follower zk_znode_count 550 <----- Total number of znodes zk_watch_count 27 zk_ephemerals_count 13 zk_approximate_data_size 1073741824 <---- Size in Bytes zk_open_file_descriptor_count 62 zk_max_file_descriptor_count 4096
– We also need to calculate the size of last snapshot as it is also stored in heap. Execute below command to get data dir for zookeeper :
# grep dataDir /etc/zookeeper/conf/zoo.cfg # du -sch /Path/to/dataDir/version-X/*
In my case size of znodes and latest snapshot is approximately 1GB. In this case bumping heap size to at least 4GB would be a good option.
One should be careful while deciding the heap size of zookeeper. Swapping should never be allowed to happen in the ZooKeeper server. So generally it is recommended that ZooKeeper servers have a reasonably high memory configured.