Kafdrop is a webUI for Apache Kafka. Kafdrop is an open-source tool which displays the information like broker details, creates a topic, delete a topic, browses messages and view ACLs. It is a lightweight tool and very easy to set up.
What is Kafka?
Apache Kafka is an open-source platform. Kafka was originally developed by Linkedin and was later incubated as the Apache Project. It can process over 1 million messages per second.
Need for Kafdrop:
Kafka is an amazing platform for processing a huge number of messages very quickly. However, Kafka has one disadvantage that it does not come with an inbuilt User Interface where the users can see the information related to Kafka.
Kafdrop helps us in solving this problem. It gives us a simple, lightweight, and easy-to-use User Interface where one can not only see the required information but can also create and delete Kafka topics.
Steps to install Kafdrop:
To install Kafdrop you need the following:
- Kafka 2.0 or later
- Zookeeper 3.4.5 or later
- Java 8
The source code for Kafdrop is available at https://github.com/obsidiandynamics/kafdrop
Kafdrop can be installed by executing a JAR file or via docker or on Kubernetes.
In this tutorial we will learn about installing Kafdrop via docker or on Kubernetes.
KafDrop Via Docker:
KafDrop Images are available at: hub.docker.com/r/obsidiandynamics/kafdrop.
To launch the Kafdrop run the following command:
docker run -d --rm -p 9000:9000 \ -e KAFKA_BROKERCONNECT=<kafka_broker_ip:port,kafka_broker_ip:port> \ -e JVM_OPTS="-Xms32M -Xmx64M" \ -e SERVER_SERVLET_CONTEXTPATH="/" \ obsidiandynamics/kafdrop
Now, you can access the Kafdrop UI by opening http://localhost:9000 in your browser.
KafDrop via K8S Manifest file:
You can also install KafDrop on the Kubernetes cluster with the helm of the manifest file. Create a YAML file called
kafdrop-deployment.yaml with the following content in it:
--- apiVersion: apps/v1 kind: Deployment metadata: name: kafka-kafdrop-deployment namespace: "kafdrop" labels: app: kafka-kafdrop spec: replicas: 1 selector: matchLabels: app: kafka-kafdrop strategy: rollingUpdate: maxSurge: 1 maxUnavailable: 0 type: RollingUpdate template: metadata: labels: app: kafka-kafdrop spec: volumes: - name: tz-config hostPath: path: /usr/share/zoneinfo/Asia/Kolkata containers: - image: obsidiandynamics/kafdrop imagePullPolicy: Always name: kafka-kafdrop volumeMounts: - name: tz-config mountPath: /etc/localtime resources: limits: cpu: 200m memory: 1Gi requests: cpu: 200m memory: 1Gi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File ports: - containerPort: 5010 name: server - containerPort: 5012 name: jmx env: - name: JVM_OPTS value: "-Xms512M -Xms512M" - name: SERVER_SERVLET_CONTEXTPATH value: "/" - name: KAFKA_BROKERCONNECT value: "<kafka_broker_ip>:9092" restartPolicy: Always
To create the deployment run the following command:
kubectl apply -f kafdrop-deployment.yaml
By default, the topic creation and deletion are enabled via KafDrop. If you want to disable topic creation and topic deletion, add the following in the env section of the YAML file:
- name: CMD_ARGS value: "--topic.deleteEnabled=false --topic.createEnabled=false"
To access the Kafdrop UI from the browser of your local machine you need to create a Kubernetes service that will point to the deployment create in the previous step. Create
kafdrop-service.yaml and add the following lines in it:
--- apiVersion: v1 kind: Service metadata: name: kafka-kafdrop-service namespace: "kafdrop" labels: app: kafka-kafdrop annotations: cloud.google.com/load-balancer-type: "Internal" spec: ports: - protocol: "TCP" port: 9000 name: server selector: app: kafka-kafdrop type: LoadBalancer
To create the service run the following command:
kubectl apply -f kafdrop-service.yaml
Wait for the service to come up. Once the service is started you can access the Kafdrop UI from the browser at http://<kubernetes_service_ip>:9000
With this your KafDrop is up and running.
Check Details of Kafka Cluster:
To check the details about the cluster in the browser, it will show the details like total topics, topic name, partitions, broker details as shown in the below image:
Check details of Kafka Topic:
On the KafDrop Home page click on any Kafka topic for which you want to check the details. It will open up a page with details like partitions count, replication factor, offset lag, under-replicated partition as shown in the below image
To read about how to create a Kafka HA cluster please refer to Kafka and Zookeeper HA Cluster Setup.