Skip to main content

Kafka operations(installation, create topic, create cluster, producer, consumer) on Cloudera quickstart vm

## Installation Kafka packages on Cloudera Quickstart with yum
[cloudera@quickstart ~]$ sudo yum install kafka
[cloudera@quickstart ~]$ sudo yum install kafka-server

# Edit /etc/kafka/conf/server.properties to ensure that the broker.id is unique for each node and broker in Kafka cluster, and zookeeper.connect points to same ZooKeeper for all nodes and brokers.
[cloudera@quickstart ~]$ vi /etc/kafka/conf/server.properties

# Start the Kafka server with the following command
[cloudera@quickstart ~]$ sudo service kafka-server start


# To verify all nodes are correctly registered to the same ZooKeeper, connect to ZooKeeper using zookeeper-client.
[cloudera@quickstart ~]$ zookeeper-client
# List kafka broker
[zk: localhost:2181(CONNECTED) 0] ls /brokers/ids
[0]
# To discover to which node a particular ID is assigned, use the following command:
[zk: localhost:2181(CONNECTED) 1] get /brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://quickstart.cloudera:9092"],"jmx_port":-1,"host":"quickstart.cloudera","timestamp":"1500371130054","port":9092,"version":4}
cZxid = 0x59a
ctime = Tue Jul 18 02:45:30 PDT 2017
mZxid = 0x59a
mtime = Tue Jul 18 02:45:30 PDT 2017
pZxid = 0x59a
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15d5487b585001a
dataLength = 208
numChildren = 0
# List topics on kafka via zookeeper
[zk: localhost:2181(CONNECTED) 2] ls /brokers/topics
[__consumer_offsets, test]


# List kafka topics
[cloudera@quickstart ~]$ kafka-topics --list --zookeeper localhost:2181

# Run the producer in terminal and enter some messages
[cloudera@quickstart ~]$ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
[cloudera@quickstart ~]$ kafka-topics --list --zookeeper localhost:2181

## Sending and receiving message
[cloudera@quickstart ~]$ kafka-console-producer --broker-list localhost:9092 --topic test
message 1
message 2

[cloudera@quickstart ~]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning
message 1
message 2




## Setting up a multi broker cluster
# Stop kafka-server service
[cloudera@quickstart ~]$ sudo service kafka-server stop

# Create config files for each new broker
[cloudera@quickstart]$ sudo cp /etc/kafka/conf/server.properties /etc/kafka/conf/server-1.properties
[cloudera@quickstart]$ sudo cp /etc/kafka/conf/server.properties /etc/kafka/conf/server-2.properties

# Update the following properties in the new files
/etc/kafka/conf/server-1.properties:
    broker.id=1
    listeners=PLAINTEXT://:9093
    log.dir=/tmp/kafka-logs-1

/etc/kafka/conf/server-2.properties:
    broker.id=2
    listeners=PLAINTEXT://:9094
    log.dir=/tmp/kafka-logs-2

# Start first node with service command
[cloudera@quickstart ~]$ sudo service kafka-server stop

# Start the two new nodes in separate termial windows
# You should already have Zookeeper running
[cloudera@quickstart]$ /usr/lib/kafka/bin/kafka-server-start.sh /etc/kafka/conf/server-1.properties &
...
[cloudera@quickstart]$ /usr/lib/kafka/bin/kafka-server-start.sh /etc/kafka/conf/server-2.properties &
...

# To verify all nodes are correctly registered to the same ZooKeeper, connect to ZooKeeper using zookeeper-client.
[cloudera@quickstart ~]$ zookeeper-client
# List kafka broker
[zk: localhost:2181(CONNECTED) 1] ls /brokers/ids
[2, 1, 0]


# Create new topic replicated to all 3 nodes
[cloudera@quickstart]$ kafka-topics --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Created topic "my-replicated-topic".

# See stats about our new topic
[cloudera@quickstart]$ kafka-topics --describe --zookeeper localhost:2181 --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Topic:my-replicated-topic    PartitionCount:1    ReplicationFactor:3    Configs:
    Topic: my-replicated-topic    Partition: 0    Leader: 0    Replicas: 1,0,2Isr: 0,2

# See stats about test topic
[cloudera@quickstart]$ kafka-topics --describe --zookeeper localhost:2181 --topic test
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Topic:test    PartitionCount:1    ReplicationFactor:1    Configs:
Topic: test    Partition: 0    Leader: 0    Replicas: 0    Isr: 0

# List all topics
[cloudera@quickstart]$ kafka-topics --list --zookeeper localhost:2181
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
__consumer_offsets
my-replicated-topic
test





## Testing fault tolerance
# Send some messages to our replicated topic, then kill the producer
[cloudera@quickstart]$ kafka-console-producer --broker-list localhost:9092 --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
message99
message100
message101
# Now kill Broker 1
message102
message103
message104
# Now kill Broker 2
message105
[2017-07-18 06:42:16,474] WARN Got error produce response with correlation id 9 on topic-partition my-replicated-topic-0, retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
# Run again Broker 1
message106
message107

# Read messages from topic
[cloudera@quickstart]$ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
message99
message100
message101
message102  # After kill Broker 1, consumer pull to message102
message103
message104
message105  # After kill Broker 2, consumer pull to message102
message106
message107




## Importing and exporting data
# Create a simple text file to work with that has 2 lines
[cloudera@quickstart]$ echo -e "foo\nbar" > /home/cloudera/DEVELOPMENT/KAFKA/test.txt

[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/connect-standalone.sh /etc/kafka/conf/connect-standalone.properties /etc/kafka/conf/connect-file-source.properties /etc/kafka/conf/connect-file-sink.properties

[cloudera@quickstart KAFKA]$ ls
example  kafka_2.12-0.11.0.0  test.sink.txt  test.txt

# Once the above connector starts running, it will read test.txt and write to test.sink.txt
# We can see this by reading the contents of the file
[cloudera@quickstart KAFKA]$ cat test.sink.txt
foo
bar

# To see the data in the consumer run the following
[cloudera@quickstart KAFKA]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic connect-test --from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}

# In a separate terminal window, add some lines to the file
[cloudera@quickstart]$ echo "new line 1" >> /home/cloudera/DEVELOPMENT/KAFKA/test.txt

[cloudera@quickstart KAFKA]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic connect-test --from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
{"schema":{"type":"string","optional":false},"payload":"new line 1"}

# Append new line to test.txt
[cloudera@quickstart]$ echo "new line 2" >> /home/cloudera/DEVELOPMENT/KAFKA/test.txt

[cloudera@quickstart KAFKA]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic connect-test --from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
{"schema":{"type":"string","optional":false},"payload":"new line 1"}
{"schema":{"type":"string","optional":false},"payload":"new line 2"}

[cloudera@quickstart KAFKA]$ cat test.txt
foo
bar
new line 1
new line 2


## Stream data processing


# Run built-in WordCount algorithm on data stream
# This demo application behaves slightly differently because it is designed to
# operate on an infinite, unbounded stream of data instead of a bound file

# Create some data in a file
[cloudera@quickstart KAFKA]$ echo -e "all streams lead to kafka\nhello kafka streams\njoin kafka summit" > file-input.txt

# Send data to new topic
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/kafka-topics.sh --create \
>             --zookeeper localhost:2181 \
>             --replication-factor 1 \
>             --partitions 1 \
>             --topic streams-file-input
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Created topic "streams-file-input".



# Send data to the topic
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic streams-file-input < file-input.txt
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]


# Run WordCount application, this will terminate after a few seconds
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/kafka-run-class.sh org.apache.kafka.streams.examples.wordcount.WordCountDemo

# Inspect the output by reading from the topic where the data was written
# Use the built-in String and Long deserializers
[cloudera@quickstart KAFKA]$  /usr/lib/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
>             --topic streams-wordcount-output \
>             --from-beginning \
>             --formatter kafka.tools.DefaultMessageFormatter \
>             --property print.key=true \
>             --property print.value=true \
>             --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
>             --property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
all    1
lead    1
to    1
hello    1
streams    2
join    1
kafka    3
summit    1

Comments

Popular posts from this blog

Offline AI Coding in VSCode: Continue + Ollama Complete Setup Guide

Want GitHub Copilot-level code assistance — but fully offline, air-gapped, and private? This guide walks you through setting up Continue (a VSCode extension) with Ollama (a local model runner) so you get chat, autocomplete, and inline editing powered entirely by models running on your own machine. No internet required at runtime. Table of Contents What Are Continue and Ollama? Prerequisites & Hardware Requirements Choosing the Right Models Step 1 — Installing Ollama & Pulling Models Step 2 — Installing Continue in VSCode (Offline) Step 3 — Configuring config.yaml Step 4 — Disabling Telemetry Step 5 — Usage & Shortcuts Troubleshooting What Are Continue and Ollama? Continue is an open-source VSCode (and JetBrains) extension that adds AI-powered coding features directly into your editor — think chat, inline edits, and tab autocomplete. By default it can connect to Claude, GPT-4, or any API. But it also supports local providers , which is exactly what we want here. Ollama i...

Local LLM on Ubuntu (CPU-Only): Ollama + Qwen Coder + Open WebUI + RAG Setup

Local LLM on Ubuntu (CPU-Only): Ollama + Qwen Coder + Open WebUI + RAG Run a private, high-performance internal AI coding assistant on your own hardware—no GPU required. Feb 13, 2026 Qwen 2.5 Coder RAG Enabled Enterprise Ready Keeping your code and documents on your own infrastructure is the gold standard for enterprise security. By combining Ollama , Qwen Coder , and Open WebUI , we can build a production-grade AI assistant that respects your privacy. Environment: CPU-only Ubuntu server (e.g., i9/Threadripper) with 64GB+ RAM for smooth 14B model inference. 1. System Preparation First, update your local environment and install necessary dependencies: sudo apt update && sudo apt upgrade -y sudo apt install -y curl git ca-certificates lscpu 2. Ollama & Model Tuning Install Ollama and optimize the configuration for CPU-only inference by setting the thread count: ...

Defining tomcat and mysql server Amazon EC2 instance

-- Create new amazon ec2 instance http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EC2_GetStarted.html -- Give permisson to pem file chmod 400 mycertificate.pem -- Connect to EC2 instance ssh -i mycertificate.pem ubuntu@ec2-51-51-51-51.eu-west-1.compute.amazonaws.com -- Installation Oracle Java 1.8 $ wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u92-b14/jdk-8u92-linux-x64.tar.gz $ tar -xzvf jdk-8u92-linux-x64.tar.gz $ sudo mkdir -p /usr/java/ $ sudo mv jdk1.8.0_92/ /usr/java/ $ sudo vim  /etc/profile         JAVA_HOME=/usr/java/jdk1.8.0_92         PATH=$PATH:$HOME/bin:$JAVA_HOME/bin         JRE_HOME=/usr/java/jdk1.8.0_92/jre         PATH=$PATH:$HOME/bin:$JRE_HOME/bin         export JAVA_HOME...