Kafka Console Commands

Kafka Console Commands are the basic CLI commands which can be used to complete the task like to create topic, alter topic, produce messages to Kafka or consume message from Kafka, etc. We can also use these Kafka commands to debug issues with Kafka.

What is Kafka?

Kafka is an open-source platform. It is a distributed streaming platform that is used to produce and consume streams of data. Kafka was originally developed by Linkedin and was later incubated as the Apache Project. It can process over 1 million messages per second.

Basic Kafka Console Commands:

Few of the basic Kafka console commands which can turn out to be really handy while working with Kafka are:

Check Kafka Broker status:

After the Kafka service gets started on the servers, Kafka brokers get registered with the zookeeper. To check if Kafka broker has registered itself with the zookeeper run the following commands on the zookeeper node:

/opt/kafka/bin/zookeeper-shell.sh localhost:2181
> ls /brokers/ids/
 [0,1,2]

The above command should give the output as shown in below image:

If any of the broker id is not present in the list then that broker is not functioning properly.

Topic Creation:

Kafka brokers store the data in Kafka topics. To create a Kafka topic run the following command on any of the Kafka nodes:

/opt/kafka/bin/kafka-topics.sh --zookeeper <server_ip>:2181 --create --topic <topic name> --partitions <count> --replication-factor <replication count value> --config retention.ms=<value in milliseconds>

The above command should give the output as shown in below image:

To describe the topic:

You can check the details about the Kafka topic using --describe option in the kafka-topics.sh file.

/opt/kafka/bin/kafka-topics.sh --zookeeper <zookeeper_ip>:2181 --describe --topic <topic name>

Once you execute the above command you can check the details like replication-factor, partitions count, the retention period, or any other extra configuration option which you have specified while creating the topic.

To alter the topic:

You can make changes to the current configuration of your Kafka topic using --alter option in the kafka-topics.sh file.

/opt/kafka/bin/kafka-topics.sh --alter --zookeeper <server-ip>:2181 --topic  <topic name> --partitions 80 --config retention.ms=3600000 

Kafka Console Producer:

To produce messages to Kafka topic we can use an inbuilt utility tool called Kafka console producer. Using Kafka console producer one can test if they are able to produce messages to Kafka, test their configurations, etc. You can run Kafka console producer using the below-mentioned command:

/opt/kafka/bin/kafka-console-producer.sh --broker-list <broker ip>:9092 --topic <topic name>

The above command will give the output as shown in the below image:

Kafka Console Consumer:

Kafka console consumer can be used to consume messages from Kafka topic using the CLI. Using Kafka console consumer one can test if they are able to consume messages from Kafka, test their configurations, etc. You can run Kafka console consumer using the below-mentioned command:

/opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server <broker ip>:9092 --topic <topic name> --from-beginning

The above command will give the output as shown in the below image:

Check if there are under replicated topics:

In case of some issues with your Kafka cluster, there are chances that one or more replicas of topics’ partitions might get out of sync with the leader. An under-replicated partition is a situation in which one or more replicas of a partition are not in sync with the leader.

/opt/kafka/bin/kafka-topics.sh --zookeeper <zookeeper_ip>:2181 --describe --under-replicated-partitions

If the output of the above command is blank it means there are no under-replicated partitions in your Kafka cluster.

To check if there are offline partitions:

If the cause for under-replicated partitions in the cluster is not fixed soon then there are chances that some of the partitions might go offline. Offline partitions happen when all the replicas of a partition are not in sync with the leader.

/opt/kafka/bin/kafka-topics.sh --zookeeper <zookeeper_ip>:2181 --describe | grep "Leader: -1"

If the output of the above command is blank it means there are no offline partitions in your Kafka cluster.

Kafka Consumer group list:

One can get the list of currently active Kafka consumer group using the following command:

/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server <kafka broker ip>:9092 --list

Kafka Consumer group:

To check the details about the Kafka consumer group you can use an inbuilt utility called Kafka consumer group. You can do that with the help of the below-mentioned command:

/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server <kafka broker ip>:9092 --describe --group <consumer_group_name>

Kafka Consumer group lag:

Kafka consumers can lag behind if the rate at which they are consuming messages is very less or due to some other reason. Using the above-mentioned command, you can also get the consumer lag per partition for a Kafka topic.

These were some basic Kafka console commands which can be very helpful for doing POC with Kafka, completing small Kafka tasks, or while debugging any Kafka-related issues.

To read more about Kafka cluster setup, refer to my article on Kafka and Zookeeper HA Cluster Setup.

Cheers !!!