kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 宋英拴 <songyingsh...@gmail.com>
Subject Re: Help - Issue with Kafka Machines
Date Mon, 03 Aug 2020 11:55:21 GMT
Hi Manoj,
The logs just seems the follower replicas of topic-partition
`psvc-global-data-0` are all out of ISR(In-Sync replica), so the ISR is 1
now.
And the config `min.insync.replicas` of this topic was set to 2, which
means the minimum number of replicas that must acknowledge a write for the
write to be considered successful. So the new record append failed beacuse
2 is larger than 1.
So you are suggested to check out the reason why the follower replicas out
of sync, maybe other brokers down etc.

Manoj Chitrala <manoj.chitrala@ve.com> 于2020年8月2日周日 上午2:14写道:

> Hi,
>
> Can anyone help with our kafka servers. Below is the service status.
>
>
> [root@d-u2-kfk-ms-03 bin]# systemctl status kafka -l
>
> ● kafka.service - Apache Kafka server (broker)
>
>    Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor
> preset: disabled)
>
>    Active: active (running) since Sat 2020-08-01 17:50:16 UTC; 21min ago
>
>      Docs: http://kafka.apache.org/documentation.html
>
>  Main PID: 106439 (java)
>
>    CGroup: /system.slice/kafka.service
>
>            └─106439 java -Xmx1G -Xms1G -server -XX:+UseG1GC
> -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
> -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true
> -Xloggc:/opt/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc
> -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.port=9990
> -Dkafka.logs.dir=/opt/kafka/bin/../logs
> -Dlog4j.configuration=file:/opt/kafka/config/log4j.properties -cp
> /opt/kafka/bin/../libs/aopalliance-repackaged-2.5.0-b32.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/commons-lang3-3.5.jar:/opt/kafka/bin/../libs/connect-api-1.1.0.jar:/opt/kafka/bin/../libs/connect-file-1.1.0.jar:/opt/kafka/bin/../libs/connect-json-1.1.0.jar:/opt/kafka/bin/../libs/connect-runtime-1.1.0.jar:/opt/kafka/bin/../libs/connect-transforms-1.1.0.jar:/opt/kafka/bin/../libs/guava-20.0.jar:/opt/kafka/bin/../libs/hk2-api-2.5.0-b32.jar:/opt/kafka/bin/../libs/hk2-locator-2.5.0-b32.jar:/opt/kafka/bin/../libs/hk2-utils-2.5.0-b32.jar:/opt/kafka/bin/../libs/jackson-annotations-2.9.4.jar:/opt/kafka/bin/../libs/jackson-core-2.9.4.jar:/opt/kafka/bin/../libs/jackson-databind-2.9.4.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.9.4.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.9.4.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.9.4.jar:/opt/kafka/bin/../libs/javassist-3.20.0-GA.jar:/opt/kafka/bin/../libs/javassist-3.21.0-GA.jar:/opt/kafka/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka/bin/../libs/javax.inject-1.jar:/opt/kafka/bin/../libs/javax.inject-2.5.0-b32.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs-api-2.0.1.jar:/opt/kafka/bin/../libs/jersey-client-2.25.1.jar:/opt/kafka/bin/../libs/jersey-common-2.25.1.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.25.1.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.25.1.jar:/opt/kafka/bin/../libs/jersey-guava-2.25.1.jar:/opt/kafka/bin/../libs/jersey-media-jaxb-2.25.1.jar:/opt/kafka/bin/../libs/jersey-server-2.25.1.jar:/opt/kafka/bin/../libs/jetty-client-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-continuation-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-http-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-io-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-security-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-server-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-servlet-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-servlets-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jetty-util-9.2.24.v20180105.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/kafka_2.11-1.1.0.jar:/opt/kafka/bin/../libs/kafka_2.11-1.1.0-sources.jar:/opt/kafka/bin/../libs/kafka_2.11-1.1.0-test-sources.jar:/opt/kafka/bin/../libs/kafka-clients-1.1.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-1.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-1.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-1.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-1.1.0.jar:/opt/kafka/bin/../libs/kafka-tools-1.1.0.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/lz4-java-1.4.jar:/opt/kafka/bin/../libs/maven-artifact-3.5.2.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka/bin/../libs/plexus-utils-3.1.0.jar:/opt/kafka/bin/../libs/reflections-0.9.11.jar:/opt/kafka/bin/../libs/rocksdbjni-5.7.3.jar:/opt/kafka/bin/../libs/scala-library-2.11.12.jar:/opt/kafka/bin/../libs/scala-logging_2.11-3.7.2.jar:/opt/kafka/bin/../libs/scala-reflect-2.11.12.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.25.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.25.jar:/opt/kafka/bin/../libs/snappy-java-1.1.7.1.jar:/opt/kafka/bin/../libs/validation-api-1.1.0.Final.jar:/opt/kafka/bin/../libs/zkclient-0.10.jar:/opt/kafka/bin/../libs/zookeeper-3.4.10.jar
> kafka.Kafka /opt/kafka/config/server.properties
>
>
>
> Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]:
> org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
> replicas for partition psvc-global-data-0 is [1], below required minimum [2]
>
> Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]: [2020-08-01
> 18:11:52,618] ERROR [ReplicaManager broker=3] Error processing append
> operation on partition psvc-global-data-0 (kafka.server.ReplicaManager)
>
> Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]:
> org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
> replicas for partition psvc-global-data-0 is [1], below required minimum [2]
>
> Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]: [2020-08-01
> 18:11:52,619] ERROR [ReplicaManager broker=3] Error processing append
> operation on partition psvc-global-data-0 (kafka.server.ReplicaManager)
>
> Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]:
> org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
> replicas for partition psvc-global-data-0 is [1], below required minimum [2]
>
> Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]: [2020-08-01
> 18:11:52,619] ERROR [ReplicaManager broker=3] Error processing append
> operation on partition psvc-global-data-0 (kafka.server.ReplicaManager)
>
> Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]:
> org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
> replicas for partition psvc-global-data-0 is [1], below required minimum [2]
>
> Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]: [2020-08-01
> 18:11:52,625] ERROR [ReplicaManager broker=3] Error processing append
> operation on partition psvc-global-data-0 (kafka.server.ReplicaManager)
>
> Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]:
> org.apache.kafka.common.errors.NotEnoughReplicasException: Number of insync
> replicas for partition psvc-global-data-0 is [1], below required minimum [2]
>
> Aug 01 18:11:52 d-u2-kfk-ms-03.veproduction.dom kafka[106439]: [2020-08-01
> 18:11:52,626] ERROR [ReplicaManager broker=3] Error processing append
> operation on partition psvc-global-data-0 (kafka.server.ReplicaManager)
>
> [root@d-u2-kfk-ms-03 bin]#
>
>
> Your help is much appreciated.
> Thanks,
> Manoj Chitrala
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message