Kafka operations(installation, create topic, create cluster, producer, consumer) on Cloudera quickstart vm
## Installation Kafka packages on Cloudera Quickstart with yum
[cloudera@quickstart ~]$ sudo yum install kafka
[cloudera@quickstart ~]$ sudo yum install kafka-server
# Edit /etc/kafka/conf/server.properties to ensure that the broker.id is unique for each node and broker in Kafka cluster, and zookeeper.connect points to same ZooKeeper for all nodes and brokers.
[cloudera@quickstart ~]$ vi /etc/kafka/conf/server.properties
# Start the Kafka server with the following command
[cloudera@quickstart ~]$ sudo service kafka-server start
# To verify all nodes are correctly registered to the same ZooKeeper, connect to ZooKeeper using zookeeper-client.
[cloudera@quickstart ~]$ zookeeper-client
# List kafka broker
[zk: localhost:2181(CONNECTED) 0] ls /brokers/ids
[0]
# To discover to which node a particular ID is assigned, use the following command:
[zk: localhost:2181(CONNECTED) 1] get /brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://quickstart.cloudera:9092"],"jmx_port":-1,"host":"quickstart.cloudera","timestamp":"1500371130054","port":9092,"version":4}
cZxid = 0x59a
ctime = Tue Jul 18 02:45:30 PDT 2017
mZxid = 0x59a
mtime = Tue Jul 18 02:45:30 PDT 2017
pZxid = 0x59a
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15d5487b585001a
dataLength = 208
numChildren = 0
# List topics on kafka via zookeeper
[zk: localhost:2181(CONNECTED) 2] ls /brokers/topics
[__consumer_offsets, test]
# List kafka topics
[cloudera@quickstart ~]$ kafka-topics --list --zookeeper localhost:2181
# Run the producer in terminal and enter some messages
[cloudera@quickstart ~]$ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
[cloudera@quickstart ~]$ kafka-topics --list --zookeeper localhost:2181
## Sending and receiving message
[cloudera@quickstart ~]$ kafka-console-producer --broker-list localhost:9092 --topic test
message 1
message 2
[cloudera@quickstart ~]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning
message 1
message 2
## Setting up a multi broker cluster
# Stop kafka-server service
[cloudera@quickstart ~]$ sudo service kafka-server stop
# Create config files for each new broker
[cloudera@quickstart]$ sudo cp /etc/kafka/conf/server.properties /etc/kafka/conf/server-1.properties
[cloudera@quickstart]$ sudo cp /etc/kafka/conf/server.properties /etc/kafka/conf/server-2.properties
# Update the following properties in the new files
/etc/kafka/conf/server-1.properties:
broker.id=1
listeners=PLAINTEXT://:9093
log.dir=/tmp/kafka-logs-1
/etc/kafka/conf/server-2.properties:
broker.id=2
listeners=PLAINTEXT://:9094
log.dir=/tmp/kafka-logs-2
# Start first node with service command
[cloudera@quickstart ~]$ sudo service kafka-server stop
# Start the two new nodes in separate termial windows
# You should already have Zookeeper running
[cloudera@quickstart]$ /usr/lib/kafka/bin/kafka-server-start.sh /etc/kafka/conf/server-1.properties &
...
[cloudera@quickstart]$ /usr/lib/kafka/bin/kafka-server-start.sh /etc/kafka/conf/server-2.properties &
...
# To verify all nodes are correctly registered to the same ZooKeeper, connect to ZooKeeper using zookeeper-client.
[cloudera@quickstart ~]$ zookeeper-client
# List kafka broker
[zk: localhost:2181(CONNECTED) 1] ls /brokers/ids
[2, 1, 0]
# Create new topic replicated to all 3 nodes
[cloudera@quickstart]$ kafka-topics --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Created topic "my-replicated-topic".
# See stats about our new topic
[cloudera@quickstart]$ kafka-topics --describe --zookeeper localhost:2181 --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 0 Replicas: 1,0,2Isr: 0,2
# See stats about test topic
[cloudera@quickstart]$ kafka-topics --describe --zookeeper localhost:2181 --topic test
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
# List all topics
[cloudera@quickstart]$ kafka-topics --list --zookeeper localhost:2181
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
__consumer_offsets
my-replicated-topic
test
## Testing fault tolerance
# Send some messages to our replicated topic, then kill the producer
[cloudera@quickstart]$ kafka-console-producer --broker-list localhost:9092 --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
message99
message100
message101
# Now kill Broker 1
message102
message103
message104
# Now kill Broker 2
message105
[2017-07-18 06:42:16,474] WARN Got error produce response with correlation id 9 on topic-partition my-replicated-topic-0, retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
# Run again Broker 1
message106
message107
# Read messages from topic
[cloudera@quickstart]$ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
message99
message100
message101
message102 # After kill Broker 1, consumer pull to message102
message103
message104
message105 # After kill Broker 2, consumer pull to message102
message106
message107
## Importing and exporting data
# Create a simple text file to work with that has 2 lines
[cloudera@quickstart]$ echo -e "foo\nbar" > /home/cloudera/DEVELOPMENT/KAFKA/test.txt
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/connect-standalone.sh /etc/kafka/conf/connect-standalone.properties /etc/kafka/conf/connect-file-source.properties /etc/kafka/conf/connect-file-sink.properties
[cloudera@quickstart KAFKA]$ ls
example kafka_2.12-0.11.0.0 test.sink.txt test.txt
# Once the above connector starts running, it will read test.txt and write to test.sink.txt
# We can see this by reading the contents of the file
[cloudera@quickstart KAFKA]$ cat test.sink.txt
foo
bar
# To see the data in the consumer run the following
[cloudera@quickstart KAFKA]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic connect-test --from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
# In a separate terminal window, add some lines to the file
[cloudera@quickstart]$ echo "new line 1" >> /home/cloudera/DEVELOPMENT/KAFKA/test.txt
[cloudera@quickstart KAFKA]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic connect-test --from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
{"schema":{"type":"string","optional":false},"payload":"new line 1"}
# Append new line to test.txt
[cloudera@quickstart]$ echo "new line 2" >> /home/cloudera/DEVELOPMENT/KAFKA/test.txt
[cloudera@quickstart KAFKA]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic connect-test --from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
{"schema":{"type":"string","optional":false},"payload":"new line 1"}
{"schema":{"type":"string","optional":false},"payload":"new line 2"}
[cloudera@quickstart KAFKA]$ cat test.txt
foo
bar
new line 1
new line 2
## Stream data processing
# Run built-in WordCount algorithm on data stream
# This demo application behaves slightly differently because it is designed to
# operate on an infinite, unbounded stream of data instead of a bound file
# Create some data in a file
[cloudera@quickstart KAFKA]$ echo -e "all streams lead to kafka\nhello kafka streams\njoin kafka summit" > file-input.txt
# Send data to new topic
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/kafka-topics.sh --create \
> --zookeeper localhost:2181 \
> --replication-factor 1 \
> --partitions 1 \
> --topic streams-file-input
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Created topic "streams-file-input".
# Send data to the topic
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic streams-file-input < file-input.txt
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
# Run WordCount application, this will terminate after a few seconds
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/kafka-run-class.sh org.apache.kafka.streams.examples.wordcount.WordCountDemo
# Inspect the output by reading from the topic where the data was written
# Use the built-in String and Long deserializers
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
> --topic streams-wordcount-output \
> --from-beginning \
> --formatter kafka.tools.DefaultMessageFormatter \
> --property print.key=true \
> --property print.value=true \
> --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
> --property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
all 1
lead 1
to 1
hello 1
streams 2
join 1
kafka 3
summit 1
[cloudera@quickstart ~]$ sudo yum install kafka
[cloudera@quickstart ~]$ sudo yum install kafka-server
# Edit /etc/kafka/conf/server.properties to ensure that the broker.id is unique for each node and broker in Kafka cluster, and zookeeper.connect points to same ZooKeeper for all nodes and brokers.
[cloudera@quickstart ~]$ vi /etc/kafka/conf/server.properties
# Start the Kafka server with the following command
[cloudera@quickstart ~]$ sudo service kafka-server start
# To verify all nodes are correctly registered to the same ZooKeeper, connect to ZooKeeper using zookeeper-client.
[cloudera@quickstart ~]$ zookeeper-client
# List kafka broker
[zk: localhost:2181(CONNECTED) 0] ls /brokers/ids
[0]
# To discover to which node a particular ID is assigned, use the following command:
[zk: localhost:2181(CONNECTED) 1] get /brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://quickstart.cloudera:9092"],"jmx_port":-1,"host":"quickstart.cloudera","timestamp":"1500371130054","port":9092,"version":4}
cZxid = 0x59a
ctime = Tue Jul 18 02:45:30 PDT 2017
mZxid = 0x59a
mtime = Tue Jul 18 02:45:30 PDT 2017
pZxid = 0x59a
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x15d5487b585001a
dataLength = 208
numChildren = 0
# List topics on kafka via zookeeper
[zk: localhost:2181(CONNECTED) 2] ls /brokers/topics
[__consumer_offsets, test]
# List kafka topics
[cloudera@quickstart ~]$ kafka-topics --list --zookeeper localhost:2181
# Run the producer in terminal and enter some messages
[cloudera@quickstart ~]$ kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
[cloudera@quickstart ~]$ kafka-topics --list --zookeeper localhost:2181
## Sending and receiving message
[cloudera@quickstart ~]$ kafka-console-producer --broker-list localhost:9092 --topic test
message 1
message 2
[cloudera@quickstart ~]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic test --from-beginning
message 1
message 2
## Setting up a multi broker cluster
# Stop kafka-server service
[cloudera@quickstart ~]$ sudo service kafka-server stop
# Create config files for each new broker
[cloudera@quickstart]$ sudo cp /etc/kafka/conf/server.properties /etc/kafka/conf/server-1.properties
[cloudera@quickstart]$ sudo cp /etc/kafka/conf/server.properties /etc/kafka/conf/server-2.properties
# Update the following properties in the new files
/etc/kafka/conf/server-1.properties:
broker.id=1
listeners=PLAINTEXT://:9093
log.dir=/tmp/kafka-logs-1
/etc/kafka/conf/server-2.properties:
broker.id=2
listeners=PLAINTEXT://:9094
log.dir=/tmp/kafka-logs-2
# Start first node with service command
[cloudera@quickstart ~]$ sudo service kafka-server stop
# Start the two new nodes in separate termial windows
# You should already have Zookeeper running
[cloudera@quickstart]$ /usr/lib/kafka/bin/kafka-server-start.sh /etc/kafka/conf/server-1.properties &
...
[cloudera@quickstart]$ /usr/lib/kafka/bin/kafka-server-start.sh /etc/kafka/conf/server-2.properties &
...
# To verify all nodes are correctly registered to the same ZooKeeper, connect to ZooKeeper using zookeeper-client.
[cloudera@quickstart ~]$ zookeeper-client
# List kafka broker
[zk: localhost:2181(CONNECTED) 1] ls /brokers/ids
[2, 1, 0]
# Create new topic replicated to all 3 nodes
[cloudera@quickstart]$ kafka-topics --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Created topic "my-replicated-topic".
# See stats about our new topic
[cloudera@quickstart]$ kafka-topics --describe --zookeeper localhost:2181 --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:
Topic: my-replicated-topic Partition: 0 Leader: 0 Replicas: 1,0,2Isr: 0,2
# See stats about test topic
[cloudera@quickstart]$ kafka-topics --describe --zookeeper localhost:2181 --topic test
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Topic:test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: test Partition: 0 Leader: 0 Replicas: 0 Isr: 0
# List all topics
[cloudera@quickstart]$ kafka-topics --list --zookeeper localhost:2181
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
__consumer_offsets
my-replicated-topic
test
## Testing fault tolerance
# Send some messages to our replicated topic, then kill the producer
[cloudera@quickstart]$ kafka-console-producer --broker-list localhost:9092 --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
message99
message100
message101
# Now kill Broker 1
message102
message103
message104
# Now kill Broker 2
message105
[2017-07-18 06:42:16,474] WARN Got error produce response with correlation id 9 on topic-partition my-replicated-topic-0, retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender)
# Run again Broker 1
message106
message107
# Read messages from topic
[cloudera@quickstart]$ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic my-replicated-topic
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
message99
message100
message101
message102 # After kill Broker 1, consumer pull to message102
message103
message104
message105 # After kill Broker 2, consumer pull to message102
message106
message107
## Importing and exporting data
# Create a simple text file to work with that has 2 lines
[cloudera@quickstart]$ echo -e "foo\nbar" > /home/cloudera/DEVELOPMENT/KAFKA/test.txt
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/connect-standalone.sh /etc/kafka/conf/connect-standalone.properties /etc/kafka/conf/connect-file-source.properties /etc/kafka/conf/connect-file-sink.properties
[cloudera@quickstart KAFKA]$ ls
example kafka_2.12-0.11.0.0 test.sink.txt test.txt
# Once the above connector starts running, it will read test.txt and write to test.sink.txt
# We can see this by reading the contents of the file
[cloudera@quickstart KAFKA]$ cat test.sink.txt
foo
bar
# To see the data in the consumer run the following
[cloudera@quickstart KAFKA]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic connect-test --from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
# In a separate terminal window, add some lines to the file
[cloudera@quickstart]$ echo "new line 1" >> /home/cloudera/DEVELOPMENT/KAFKA/test.txt
[cloudera@quickstart KAFKA]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic connect-test --from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
{"schema":{"type":"string","optional":false},"payload":"new line 1"}
# Append new line to test.txt
[cloudera@quickstart]$ echo "new line 2" >> /home/cloudera/DEVELOPMENT/KAFKA/test.txt
[cloudera@quickstart KAFKA]$ kafka-console-consumer --bootstrap-server localhost:9092 --topic connect-test --from-beginning
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{"schema":{"type":"string","optional":false},"payload":"foo"}
{"schema":{"type":"string","optional":false},"payload":"bar"}
{"schema":{"type":"string","optional":false},"payload":"new line 1"}
{"schema":{"type":"string","optional":false},"payload":"new line 2"}
[cloudera@quickstart KAFKA]$ cat test.txt
foo
bar
new line 1
new line 2
## Stream data processing
# Run built-in WordCount algorithm on data stream
# This demo application behaves slightly differently because it is designed to
# operate on an infinite, unbounded stream of data instead of a bound file
# Create some data in a file
[cloudera@quickstart KAFKA]$ echo -e "all streams lead to kafka\nhello kafka streams\njoin kafka summit" > file-input.txt
# Send data to new topic
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/kafka-topics.sh --create \
> --zookeeper localhost:2181 \
> --replication-factor 1 \
> --partitions 1 \
> --topic streams-file-input
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Created topic "streams-file-input".
# Send data to the topic
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic streams-file-input < file-input.txt
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
# Run WordCount application, this will terminate after a few seconds
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/kafka-run-class.sh org.apache.kafka.streams.examples.wordcount.WordCountDemo
# Inspect the output by reading from the topic where the data was written
# Use the built-in String and Long deserializers
[cloudera@quickstart KAFKA]$ /usr/lib/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
> --topic streams-wordcount-output \
> --from-beginning \
> --formatter kafka.tools.DefaultMessageFormatter \
> --property print.key=true \
> --property print.value=true \
> --property key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
> --property value.deserializer=org.apache.kafka.common.serialization.LongDeserializer
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/kafka/libs/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
all 1
lead 1
to 1
hello 1
streams 2
join 1
kafka 3
summit 1
Comments
Post a Comment