Troubleshooting kerberos issues in kafka for producer/consumers

In kafka there are three types of communication :
1. Between brokers
2. Between client and broker.
3. Broker to zookeeper

In order to communicate in kerberos enabled cluster one needs to authenticate itself. So when broker will try to communicate with other broker in the cluster it will need to authenticate first. Same for the clients communicating to the brokers.

In Kafka, JAAS files is used for authentication. Lets first understand what’s there in JAAS file. In kafka there are two JAAS files :
1. kafka_jaas.conf
2. kafka_client_jaas.conf.

Let’s discuss about kafka_jaas.conf. This file will be used for authentication when a broker in a cluster tries to communicate with other brokers in cluster. Take a look at the content and understand its purpose :

KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="kafka/c6401.ambari.apache.org@EXAMPLE.COM";
};

KafkaServer section will be used by broker for authentication when it tries to communicate with other brokers in cluster. It should always be configured to use keytab and principal.
Value of `serviceName` should be the principal as which kafka is running.

Client { // used for zookeeper connection
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
storeKey=true
useTicketCache=false
serviceName="zookeeper"
principal="kafka/c6401.ambari.apache.org@EXAMPLE.COM";
};

Client section in kafka_jaas.conf will be used for authentication when broker wants to communicate with zookeeper. It should always be configured to use keytab and principal.
Value of `serviceName` should be the principal as which zookeeper service is running.

kafka_client_jaas.conf will be used by clients(producer/consumer) to authenticate to kafka broker. kafka_client_jaas.conf has two sections, take a look :

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="kafka";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="zookeeper";
};

KafkaClient section : As the name suggest it will be used when client wants to communicate to broker.
Value of `serviceName` should be the principal as which kafka is running. You can configure it to use ticket cache or keytab and principal.

Client : This part of JAAS file is only used by clients which are using old consumer api. In old consumer api, consumers need to connect to zookeeper.

In new consumer api clients communicate to brokers instead of zookeeper. Hence while authentication it will use KafkaClient section in kafka_client_jaas.conf.

Producers will always use KafkaClient section in kafka_client_jaas.conf as it will send request to broker node.

For long running kafka clients it recommended to configure JAAS file to use keytab and principal. Please refer below example :

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/storm.service.keytab"
storeKey=true
useTicketCache=false
serviceName="kafka"
principal="storm@EXAMPLE.COM";
};

When kerberos is integrated with kafka we see lot of issues while trying to produce/consume messages to kafka. There are instances where client throws a generic error and we don’nt know what’s going wrong. To tackle such conditions I will discuss about the checks which need to be done when you face such issue :

1. make sure kerberos client is installed on the node.

2. Check if you can obtain a ticket of the principal :

– If you want to obtain TGT using password :

# kinit <principalName>

If you are using keytab check if user has permission to read the keytab :
– Using keytab :

# kinit -kt /Path/to/Keytab <PrincipalName>

3. Confirm if you a ticket in ticket cache which is not expired :

# klist

4. Confirm if user has read permission on JAAS file which is used.

5. Confirm if client can communciate to kafka broker and port on which broker is listening :

# ping <broker.hostname>
# telnet broker.hostname:port

6. Check if you are using correct security protocol `–security-protocol` as configured in server.properties in broker.

7. Try exporting JAAS file and run producer/consumer again :

export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/Path/to/Jaas/File"

In order to enable debug for kerberos :

export KAFKA_CLIENT_KERBEROS_PARAMS="-Djava.security.auth.login.config=/Path/to/Jaas/File -Dsun.security.krb5.debug=true"

8. Still if you are facing authentication issue, try enabling debug for console-producer/console-consumer on kafka client node :
As root user :

# vim /usr/hdp/current/kafka-broker/config/tools-log4j.properties :
log4j.rootLogger=DEBUG, stderr

In debug logs you should see which principal, security protocol is used and to which broker request is being send.

9. Once you are confirmed that authentication is working fine it time for you to confirm if user has the required permission on the topic.
If kafka is configured to use kafka acl’s, please refer below link :
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Authorization+Command+Line+Interface

If kafka is configured to use ranger, make sure policy is defined for the topic and principal

Leave a Reply

Your email address will not be published. Required fields are marked *