Kafka CLI Command Cheat Sheet

This article, Helps you to know the Kafka CLI Command used to create and list of topics and to start a console consumer and test it out.

List Topics

– To Check if all the topics in a cluster have a valid leader and follower

kafka-topics --describe -bootstrap-server <Broker Host:port>

Create a Topic

– To create a topic with replication and partitions

kafka-topics -bootstrap-server <Broker Host:port>—replication-factor 3 --partitions 3 --topic test

The above command will create a topic with 2 replication factors and 3 partitions.

For Secure cluster:

You have to do some extra steps to create a topic

-> Destroy existing ticket using “kdestroy”

-> kinit -kt <keytab-file-complete-path> <principal used for the user>

-> Check ticket details using klist

-> Create jaas.conf file in the following format

-> Edit <Path to >/jaas.conf

		KafkaClient {
		com.sun.security.auth.module.Krb5LoginModule required
		useKeyTab=true
		keyTab="<Keytab path>/User.keytab"
		storeKey=true
		useTicketCache=false
		renewTicket=true
		principal="user@<realm>";
		serviceName="kafka";
		};
		Client {
		com.sun.security.auth.module.Krb5LoginModule required
		useKeyTab=true
		keyTab=“<Keytab path>/User.keytab"
		storeKey=true
		useTicketCache=false
		serviceName="zookeeper"
		principal=user@<realm>”;
		};

Note: Replace your keytab path and principal

-> export KAFKA_OPTS=”-Djava.security.auth.login.config=/<Path to>/jaas.conf”

Note: Replace jaas.conf file path

-> Create client /<Path to >/client-ssl.properties

		security.protocol=SASL_SSL
		ssl.truststore.location=/<path to >/client.truststore.jks
		ssl.truststore.password=password
		sasl.kerberos.service.name=kafka

Note: Replace truststore file path and password.

-> Try to create a topic as below

kafka-topics -bootstrap-server <Broker Host:port>—replication-factor 3 --partitions 3 --topic test

Now, Let’s see, How to create a console consumer and producer with the example:

Console Consumer & Producer:

Ideally, We will use this console consumer and producer to make sure things are working fine in the Kafka cluster.

Let’s say your actual consumer (external) is unable to consume the messages from your Kafka cluster and to troubleshoot this issue, we can use this console consumer to test if it is a issue with produce or consumer.

– To start a console consumer (This help to validate if the topic or broker is working properly)

-> Open 2 terminal:

– Run the below commands in 2 terminal

Start the producer

kafka-console-producer --broker-list <Broker Hostname>:<port> --topic test

Start the Consumer

kafka-console-consumer --bootstrap-server <Broker Hostname>:<port> --topic test 

Once you have started both consumer and producer, We can try typing the texts in the producer, and will be seen in the consumer console 🙂

Hope this article is helpful for you, Good luck with your Learning !!!