kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Bueche <cbue...@gmail.com>
Subject Re: broken cluster / com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath
Date Fri, 06 Dec 2019 10:23:11 GMT
Hello Ismael,

my installations of Zookeeper and Kafka are independent, before and
after the upgrade. My previous situation was :

Zookeeper 3.4.13 and Kafka 2.12-2.1.0

I wanted to update to the latest versions (sounds not unusual) and had
this plan :

- Zookeeper 3.4.13 --> 3.5.6
- Kafka 2.12-2.1.0 --> 2.12-2.3.0

So on the broken cluster I run Zookeeper 3.5.6 with Kafka 2.12-2.3.0.

Not sure to understand the question precisely. I start zookeeper from
its /appl/zookeeper dir, and Kafka from /appl/kafka. I ran a ps and see
this in the command line kafka:

........:/appl/kafka/bin/../libs/zookeeper-3.4.14.jar:.........

so I guess Kafka uses its own zookeeper lib 3.4.14 and talks to the
Zookeeper server which itself is running 3.5.6. Do you think it could be
a reason for the issue ?

I can try to downgrade ZK and see how it goes.

Regs,
Charles

Le 05.12.19 à 17:33, Ismael Juma a écrit :
> Are you using ZooKeeper 3.5.6 client libraries with Kafka 2.3.1? Kafka
> 2.3.1 ships with ZooKeeper 3.4.x.
>
> Ismael
>
> On Thu, Dec 5, 2019 at 8:18 AM Charles Bueche <cbueche@gmail.com> wrote:
>
>> Hello all,
>>
>> I do have two Kafka clusters in action, test and prod. The two are formed
>> by 3 nodes each, are independent and run their own zookeeper setups. My
>> prod cluster is running fine. My test cluster is half-broken and I'm
>> struggling to fix it. I could wipe it but I prefer to understand what's
>> wrong and fix it.
>>
>> I'm not sure what broke my test cluster. I had several network
>> disconnections / split-brains but Kafka always recovered fine. The reasons
>> for the network issues are independent and still being investigated
>> (layer-2 storms, etc).
>>
>> So I upgraded my zookeeper and kafka to the latest versions and when
>> trying to rebalance a topic across brokers I started to notice the
>> problems. Not sure really when they started, before or after the upgrade.
>>
>> I ran the upgrade as for the official doc (rolling upgrade, moving up
>> inter.broker.protocol.version and log.message.format.version gradually).
>>
>> ------------------------------------------------------
>> cat /etc/os-release
>> NAME="SLES"
>> VERSION="12-SP4"
>> VERSION_ID="12.4"
>> PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4"
>> ID="sles"
>> ANSI_COLOR="0;32"
>> CPE_NAME="cpe:/o:suse:sles:12:sp4"
>>
>> ------------------------------------------------------
>> versions :
>> - zookeeper 3.5.6
>> - kafka 2.12-2.3.1
>>
>> ------------------------------------------------------
>>
>> many rapid log entries on brokers 1 & 2 (we have 1, 2, 3)
>>
>> [2019-12-05 09:56:54,967] ERROR [KafkaApi-1] Number of alive brokers '0'
>> does not meet the required replication factor '1' for the offsets topic
>> (configured via 'offsets.topic.replication.factor'). This error can be
>> ignored if the cluster is starting up and not all brokers are up yet.
>> (kafka.server.KafkaApis)
>>
>> ------------------------------------------------------
>> java is java-1_8_0-openjdk-1.8.0.222-27.35.2.x86_64 from SLES. I have
>> tried Oracle Java jdk1.8.0_231 with the same issue.
>>
>> ------------------------------------------------------
>> when trying to see a reassignment I have this very suspect error :
>>
>> root@vmgato701a01:/appl/kafka/bin # ./kafka-reassign-partitions.sh
>> --zookeeper $ZKLIST --reassignment-json-file /tmp/r7.json --verify
>>
>> Status of partition reassignment:
>> Partitions reassignment failed due to
>> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
>> java.lang.AbstractMethodError:
>> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
>>         at
>> com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findDeserializer(OptionalHandlerFactory.java:122)
>>         at
>> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findOptionalStdDeserializer(BasicDeserializerFactory.java:1589)
>>         at
>> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1812)
>>         at
>> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:161)
>>         at
>> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:125)
>>         at
>> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411)
>>         at
>> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
>>         at
>> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
>>         at
>> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
>>         at
>> com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
>>         at
>> com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
>>         at
>> com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178)
>>         at
>> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
>>         at
>> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3079)
>>         at kafka.utils.Json$.parseBytesAs(Json.scala:73)
>>         at kafka.zk.ReassignPartitionsZNode$.decode(ZkData.scala:407)
>>         at
>> kafka.zk.KafkaZkClient.getPartitionReassignment(KafkaZkClient.scala:795)
>>         at
>> kafka.admin.ReassignPartitionsCommand$.checkIfPartitionReassignmentSucceeded(ReassignPartitionsCommand.scala:355)
>>         at
>> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:97)
>>         at
>> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90)
>>         at
>> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
>>         at
>> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
>>
>> Help appreciated.
>>

Mime
View raw message