<>Kafka common problem

<> abstract

This paper records kafka-0.8.2.1,Kafka-1.1.1,Kafka-2.x Common problems and solutions of version .

<>1 Kafka-0.8.2.1

<>1.1 Consumer report errors

* Problem description : WARN [
console-consumer-23149_hadoop2-1476872289945-6edaec5e-leader-finder-thread],
Failed to add leaderfor partitions [filelog_v3_json_test_1019,13],....; will
retry(kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)WARN [
console-consumer-23149_hadoop2-1476872289945-6edaec5e-leader-finder-thread],
Failed to add leaderfor partitions [filelog_v3_json_test_1019,13],....; will
retry(kafka.consumer.ConsumerFetcherManager$LeaderFinderThread)
java.nio.channels.ClosedChannelException[2016-10-19 18:18:11,801] WARN Fetching
topic metadata with correlationid 4 for topics [Set(filelog_v3_json_test_1019)]
from broker[id:0,host:es-gangw,port:9092] failed (kafka.client.ClientUtils$)
* Solution :
inspect server.properties file Of host.name field , Is the configuration effective ?( Pay attention to the front # Number , It takes effect only if it is removed )
<>1.2 First push topic Error in data , It's normal to push again

* Problem description : bin/kafka-console-producer.sh --broker-list
10.1.17.51:9092,10.1.17.52:9092,10.1.17.53:9092 --topic chengc_1104 222[
2016-11-04 10:45:43,274] WARN Error while fetching metadata [{TopicMetadata for
topic chengc_1104 -> No partition metadata for topic chengc_1104 due to
kafka.common.LeaderNotAvailableException}] for topic [chengc_1104]: class
kafka.common.LeaderNotAvailableException(kafka.producer.BrokerPartitionInfo) [
2016-11-04 10:45:43,291] WARN Error while fetching metadata [{TopicMetadata for
topic chengc_1104 -> No partition metadata for topic chengc_1104 due to
kafka.common.LeaderNotAvailableException}] for topic [chengc_1104]: class
kafka.common.LeaderNotAvailableException(kafka.producer.BrokerPartitionInfo) [
2016-11-04 10:45:43,293] ERROR Failed to collate messages by topic, partition
due to: Failed to fetch topic metadatafor topic: chengc_1104 (
kafka.producer.async.DefaultEventHandler) [2016-11-04 10:45:43,402] WARN Error
while fetching metadata [{TopicMetadata for topic chengc_1104 -> No partition
metadatafor topic chengc_1104 due to kafka.common.LeaderNotAvailableException}]
for topic [chengc_1104]: class kafka.common.LeaderNotAvailableException (
kafka.producer.BrokerPartitionInfo) [2016-11-04 10:45:43,407] WARN Error while
fetching metadata[{TopicMetadata for topic chengc_1104 -> No partition metadata
for topic chengc_1104 due to kafka.common.LeaderNotAvailableException}] for
topic[chengc_1104]: class kafka.common.LeaderNotAvailableException (
kafka.producer.BrokerPartitionInfo) [2016-11-04 10:45:43,407] ERROR Failed to
collate messages by topic, partition due to: Failed to fetch topic metadatafor
topic: chengc_1104(kafka.producer.async.DefaultEventHandler) [2016-11-04
10:45:43,521] WARN Error while fetching metadata [{TopicMetadata for topic
chengc_1104 -> No partition metadata for topic chengc_1104 due to
kafka.common.LeaderNotAvailableException}] for topic [chengc_1104]: class
kafka.common.LeaderNotAvailableException(kafka.producer.BrokerPartitionInfo) [
2016-11-04 10:45:43,527] WARN Error while fetching metadata [{TopicMetadata for
topic chengc_1104 -> No partition metadata for topic chengc_1104 due to
kafka.common.LeaderNotAvailableException}] for topic [chengc_1104]: class
kafka.common.LeaderNotAvailableException(kafka.producer.BrokerPartitionInfo) [
2016-11-04 10:45:43,527] ERROR Failed to collate messages by topic, partition
due to: Failed to fetch topic metadatafor topic: chengc_1104 (
kafka.producer.async.DefaultEventHandler)
* Solution :
You need to create it manually first topic Push data again ,0.8.2.1 This is a problem
<>1.3 because Kafka auto rebalance , Causes the disk IO Hit full , Network traffic speed limit exceeds disk IO

*
Problem description
Kafka In both cases, the number of copies will be true or partition unbalanced , This leads to auto rebalance:
1.Kafka Replica synchronization timeout
2. There is a node hang point or restart

*
Solution
In this case, the shutdown can be considered rebalance : auto.leader.rebalance.enable
, Then perform the command line manual balance, You can choose your own time , Otherwise, it may lead to a very busy time rebalance Cause the load to be too high .

*
About manual self motion rebalance explain :

Whenever one broker Close or leader When the mechanism fails , On top of it topic Of partition Copies are transferred to other nodes . This means that , By default, when broker It will only be used as the partition Copy of follower existence , That is to say, client read and write requests will not be processed .

To avoid this ,Kafka There is a priority copy (preferred replicas
) The concept of . for instance , Some partition A copy of exists in 1,5,9 node , here 1 Node No leader Preferred copy of , Because he joined the replica list earlier .

You can make Kafka The cluster attempts to restore the leader of the restored replica by running the following command :
bin/kafka-preferred-replica-election.sh --zookeeper zk_host:port/chroot
<>1.4 topic Of leader All for none

* Solution
Number of copies checked factor Is it greater than the number of nodes , For example, only a single node is set factor by 2. This time it was changed to 1 Just fine
<>2 Kafka-1.1.1

<>2.1 kafka-connect

Launch official example :
./bin/connect-standalone.sh config/connect-standalone.properties
config/connect-file-source.properties config/connect-file-sink.properties&
The error report is as follows :
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.
collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
This problem has been investigated for a long time , For a long time, there was no clear solution , Now let me talk about my solution .

First of all, the reason for this problem is because OS It's already configured classpath Lower version included (18.0) Of guava package , It's not what we expected $KAFKA_HOME/libs/guava-20.0.jar, So it leads to an error .

see connect-standalone.sh It turns out that he's calling kafka-run-class.sh, And it's used in this CLASSPATH
, So we add a line directly to the front :CLASSPATH="/home/kafka/kafka_2.11-1.1.1/libs/*".

Restart connect-standalone.sh, It's all right .

Another idea is to use what we're going to use Kafka Of lib Path added to OS Of classpath in , Pay attention to the front , Priority loading .

<>2.2 Timeout of 60000ms expired before the position for partition xxx_topic
could be determined

This problem is generally due to Kafka One of the clusters Broker Node load is too high , Unable to respond in time producer The request to send data results in .

<>3 Kafka-2.x

<>2.1 Source code gradle initialization Could not find method scala

I started from github
clone Is the latest code , Version is 2.7, then gradle( The version is up to date 6.5) Keep reporting mistakes , Say there's a what Configuration Class not found .

Then the gradle Demote to 5.6.4, The error was solved .

But there are new mistakes :

Could not find method scala() for arguments [
build_2ofsm9wzcvhxxkvqblsz8ogy5$_run_closure5$_closure73$_closure105@729633ff]
on object of type org.gradle.api.plugins.scala.ScalaPlugin.
There's no answer on the Internet , Finally, I tried to demote kafka edition , Specifically, it is to branch from 2.7 And for the sake of 2.3, There was no mistake ...

Technology
©2020 ioDraw All rights reserved
Children programming children's learning route Android Development — Display food information according to customer budget Some views of a student on Hongmeng system LeetCode quick get start ① —— Array series ( Often asked in an interview , Suggested collection )error: (-215:Assertion failed) Solution Using function to realize the exchange of two numbers (C language ) realization LRU Cache(java edition ) Advanced programmer - Deep understanding of data structure SK Hynix World Premiere DDR5 Memory : The frequency goes up 5600MHz, Capacity up to 256GB Programmer's promotion 28 Sorting code of Guan 3