kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Bueche <cbue...@gmail.com>
Subject broken cluster / com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath
Date Thu, 05 Dec 2019 09:16:50 GMT
Hello all,

I do have two Kafka clusters in action, test and prod. The two are formed by 3 nodes each,
are independent and run their own zookeeper setups. My prod cluster is running fine. My test
cluster is half-broken and I'm struggling to fix it. I could wipe it but I prefer to understand
what's wrong and fix it.

I'm not sure what broke my test cluster. I had several network disconnections / split-brains
but Kafka always recovered fine. The reasons for the network issues are independent and still
being investigated (layer-2 storms, etc).

So I upgraded my zookeeper and kafka to the latest versions and when trying to rebalance a
topic across brokers I started to notice the problems. Not sure really when they started,
before or after the upgrade.

I ran the upgrade as for the official doc (rolling upgrade, moving up inter.broker.protocol.version
and log.message.format.version gradually).

------------------------------------------------------
cat /etc/os-release 
NAME="SLES"
VERSION="12-SP4"
VERSION_ID="12.4"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp4"

------------------------------------------------------
versions :
- zookeeper 3.5.6
- kafka 2.12-2.3.1

------------------------------------------------------

many rapid log entries on brokers 1 & 2 (we have 1, 2, 3)

[2019-12-05 09:56:54,967] ERROR [KafkaApi-1] Number of alive brokers '0' does not meet the
required replication factor '1' for the offsets topic (configured via 'offsets.topic.replication.factor').
This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

------------------------------------------------------
java is java-1_8_0-openjdk-1.8.0.222-27.35.2.x86_64 from SLES. I have tried Oracle Java jdk1.8.0_231
with the same issue.

------------------------------------------------------
when trying to see a reassignment I have this very suspect error :

root@vmgato701a01:/appl/kafka/bin # ./kafka-reassign-partitions.sh --zookeeper $ZKLIST --reassignment-json-file
/tmp/r7.json --verify

Status of partition reassignment: 
Partitions reassignment failed due to com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
java.lang.AbstractMethodError: com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
        at com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findDeserializer(OptionalHandlerFactory.java:122)
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findOptionalStdDeserializer(BasicDeserializerFactory.java:1589)
        at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1812)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:161)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:125)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
        at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
        at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
        at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
        at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3079)
        at kafka.utils.Json$.parseBytesAs(Json.scala:73)
        at kafka.zk.ReassignPartitionsZNode$.decode(ZkData.scala:407)
        at kafka.zk.KafkaZkClient.getPartitionReassignment(KafkaZkClient.scala:795)
        at kafka.admin.ReassignPartitionsCommand$.checkIfPartitionReassignmentSucceeded(ReassignPartitionsCommand.scala:355)
        at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:97)
        at kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90)
        at kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
        at kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)

Help appreciated.

Mime
View raw message