Apache Kafka is a distributed streaming platform. It is useful for building real-time streaming data pipelines to get data between the systems or applications. Another useful feature is real-time streaming applications that can transform streams of data or react on a stream of data

Step 1 – Prerequisites

Apache Kafka required Java to run. You must have java installed on your system. Execute below command to install default OpenJDK on your system from the official PPA’s. You can also install the specific version of from here.

sudo apt update

sudo apt install default-jdk

Step 2 – Download Apache Kafka

Download the Apache Kafka binary files from its official download website. You can also select any nearby mirror to download.

wget http://www-us.apache.org/dist/kafka/1.0.1/kafka_2.12-1.0.1.tgz

Then extract the archive file

tar xzf kafka_2.12-1.0.1.tgz

mv kafka_2.12-1.0.1 /usr/local/kafka

Step 3 – Start Kafka Server

Kafka uses ZooKeeper, so first, start a ZooKeeper server on your system. You can use the script available with Kafka to get start single-node ZooKeeper instance.

cd /usr/local/kafka

bin/zookeeper-server-start.sh config/zookeeper.properties

Now start the Kafka server:

bin/kafka-server-start.sh config/server.properties

…

[2018-03-13 10:47:45,989] INFO Kafka version : 1.0.1 (org.apache.kafka.common.utils.AppInfoParser)

[2018-03-13 10:47:45,995] INFO Kafka commitId : c0518aa65f25317e (org.apache.kafka.common.utils.AppInfoParser)

# Start Zookeeper

zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties

# Start Kafka server

kafka-server-start.sh $KAFKA_HOME/config/server.properties

# CEATE A KAFKA TOPIC called first_kafka_topic

kafka-topics.sh –create –zookeeper localhost:2181 \

–replication-factor 1 \

–partitions 1 \

–topic first_kafka_topic

# KAFKA TOPICS LIST

# Port 2181 for zookeeper is set in the zookeeper.properties file

kafka-topics.sh –list –zookeeper localhost:2181

# KAFKA PRODUCER

kafka-console-producer.sh –broker-list localhost:9092 \

–topic first_kafka_topic

# KAFKA CONSUMER

kafka-console-consumer.sh –bootstrap-server localhost:9092 \

–topic first_kafka_topic \

–from-beginning

# KAFKA DELETE A TOPIC

kafka-topics.sh –delete –zookeeper localhost:2181 –topic first_kafka_topic