Kafka and ZooKeeper contains Podman

Introduction

Apache Kafka and ZooKeeper is a distributed data store with optimization for ingesting and processing streaming data. Streaming data generated thousands. A streaming platform needs to handle this constant influx of data sequentially and incrementally process the data. Visit Apache Kafka’s site for more info. Also, try the tutorial Apache Kafka WebUI for those who want Web interface.

Kafka and ZooKeeper provide main functions to their users:

• Publish and subscribe the streams of records
• Effectively store streams data records in the order in which they were generated
• Process streams of records in real-time

Kafka primarily builds real-time streaming data pipelines and applications that adapt to the data streams. It combines messaging, storage, and stream processing to allow storage and analysis of historical and real-time data. 

Zookeeper is distributed systems configuration managing tool. It delivers multiple features for broadcast applications like distributed configuration management, leader election, consensus handling, coordination, locks, etc. Zookeeper stores its data on tree-like structures. It was introduced as Data Tree, and nodes were presented as nodes. Zookeeper follows standard Unix notations for file paths.

Pull Kafka and Zookeeper images

Use pull command from podman options.

# podman pull bitnami/kafka
# podman pull bitnami/zookeeper

Create zookeeper container pod

# podman pod create --name kafkapp -p 2181:2181 -p 9092:9092 --network bridge

Launch the Zookeeper server instance

Use the --pod kafkapp argument to the podman run command to attach the Zookeeper container to the network from the pod bridge.

# podman run --pod kafkapp --name zookeeper-server -e ALLOW_ANONYMOUS_LOGIN=yes -d bitnami/zookeeper:latest

Launch the Apache Kafka server instance

Use the --pod kafkapp argument to the podman run command to attach the Apache Kafka container to the network from the pod bridge.

# podman run --pod kafkapp --name kafka-server -e ALLOW_PLAINTEXT_LISTENER=yes -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 -d bitnami/kafka:latest

Launch your Apache Kafka and ZooKeeper client instance

Finally, we create a new container instance to launch the Apache Kafka client and connect to the server created in the previous step:

# podman run -it --rm --pod kafkapp -e KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181 bitnami/kafka:latest kafka-topics.sh --list  --bootstrap-server kafka-server:9092