kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ismael Juma <ism...@juma.me.uk>
Subject Re: broken cluster / com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath
Date Thu, 05 Dec 2019 16:33:24 GMT
Are you using ZooKeeper 3.5.6 client libraries with Kafka 2.3.1? Kafka
2.3.1 ships with ZooKeeper 3.4.x.

Ismael

On Thu, Dec 5, 2019 at 8:18 AM Charles Bueche <cbueche@gmail.com> wrote:

> Hello all,
>
> I do have two Kafka clusters in action, test and prod. The two are formed
> by 3 nodes each, are independent and run their own zookeeper setups. My
> prod cluster is running fine. My test cluster is half-broken and I'm
> struggling to fix it. I could wipe it but I prefer to understand what's
> wrong and fix it.
>
> I'm not sure what broke my test cluster. I had several network
> disconnections / split-brains but Kafka always recovered fine. The reasons
> for the network issues are independent and still being investigated
> (layer-2 storms, etc).
>
> So I upgraded my zookeeper and kafka to the latest versions and when
> trying to rebalance a topic across brokers I started to notice the
> problems. Not sure really when they started, before or after the upgrade.
>
> I ran the upgrade as for the official doc (rolling upgrade, moving up
> inter.broker.protocol.version and log.message.format.version gradually).
>
> ------------------------------------------------------
> cat /etc/os-release
> NAME="SLES"
> VERSION="12-SP4"
> VERSION_ID="12.4"
> PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4"
> ID="sles"
> ANSI_COLOR="0;32"
> CPE_NAME="cpe:/o:suse:sles:12:sp4"
>
> ------------------------------------------------------
> versions :
> - zookeeper 3.5.6
> - kafka 2.12-2.3.1
>
> ------------------------------------------------------
>
> many rapid log entries on brokers 1 & 2 (we have 1, 2, 3)
>
> [2019-12-05 09:56:54,967] ERROR [KafkaApi-1] Number of alive brokers '0'
> does not meet the required replication factor '1' for the offsets topic
> (configured via 'offsets.topic.replication.factor'). This error can be
> ignored if the cluster is starting up and not all brokers are up yet.
> (kafka.server.KafkaApis)
>
> ------------------------------------------------------
> java is java-1_8_0-openjdk-1.8.0.222-27.35.2.x86_64 from SLES. I have
> tried Oracle Java jdk1.8.0_231 with the same issue.
>
> ------------------------------------------------------
> when trying to see a reassignment I have this very suspect error :
>
> root@vmgato701a01:/appl/kafka/bin # ./kafka-reassign-partitions.sh
> --zookeeper $ZKLIST --reassignment-json-file /tmp/r7.json --verify
>
> Status of partition reassignment:
> Partitions reassignment failed due to
> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
> java.lang.AbstractMethodError:
> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
>         at
> com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findDeserializer(OptionalHandlerFactory.java:122)
>         at
> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findOptionalStdDeserializer(BasicDeserializerFactory.java:1589)
>         at
> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1812)
>         at
> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:161)
>         at
> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:125)
>         at
> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411)
>         at
> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
>         at
> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
>         at
> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
>         at
> com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
>         at
> com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
>         at
> com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178)
>         at
> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
>         at
> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3079)
>         at kafka.utils.Json$.parseBytesAs(Json.scala:73)
>         at kafka.zk.ReassignPartitionsZNode$.decode(ZkData.scala:407)
>         at
> kafka.zk.KafkaZkClient.getPartitionReassignment(KafkaZkClient.scala:795)
>         at
> kafka.admin.ReassignPartitionsCommand$.checkIfPartitionReassignmentSucceeded(ReassignPartitionsCommand.scala:355)
>         at
> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:97)
>         at
> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90)
>         at
> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
>         at
> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
>
> Help appreciated.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message