kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Bueche <cbli...@bueche.ch>
Subject Re: broken cluster / com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath
Date Wed, 11 Dec 2019 15:58:03 GMT
Hello again,

Problem is still present, something is badly broken with the handling of
JSON files. Details :

I have made multiple downgrades and rolling-upgrades, and the situation
is now :

  * Zookeeper 3.4.14
  * Kafka 2.12-2.3.1 (contains libs/zookeeper-3.4.14.jar)

The cluster is all "happy", with balanced, no skew, etc. I have
kafka-manager and everything is ok.

I then create a "testing" topic :

    export ZKLIST="srv1:2181,srv2:2181,srv3:2181"
    /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --create
--replication-factor 3 --partitions 1 --topic testing

I can rise the partitions fine :

    /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --alter --topic
testing --partitions 3                          

It then looks ok :

    /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --describe
--topic testing                                      
    Topic:testing   PartitionCount:3        ReplicationFactor:3     Configs:
        Topic: testing  Partition: 0    Leader: 1       Replicas: 1,2,3
Isr: 1,2,3
        Topic: testing  Partition: 1    Leader: 2       Replicas: 2,3,1
Isr: 2,3,1
        Topic: testing  Partition: 2    Leader: 3       Replicas: 3,1,2
Isr: 3,1,2

Attempting any operation involving a json file breaks, when  it worked
fine on Kafka 2.12-2.1.0. I shall precise my json files are passing lint
tests and work ok.

Helped needed !!!

Thanks,
Charles

/appl/kafka/bin/kafka-reassign-partitions.sh --zookeeper $ZKLIST
--reassignment-json-file ~buec/tmp/r7.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"testing","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"testing","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"testing","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Partitions reassignment failed due to
com.fasterxml.jackson.databind.ext.Java7Support.getSerializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonSerializer;
java.lang.AbstractMethodError:
com.fasterxml.jackson.databind.ext.Java7Support.getSerializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonSerializer;
        at
com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findSerializer(OptionalHandlerFactory.java:92)
        at
com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findOptionalStdSerializer(BasicSerializerFactory.java:439)
        at
com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByPrimaryType(BasicSerializerFactory.java:374)
        at
com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:226)
        at
com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)
        at
com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1385)
        at
com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1336)
        at
com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:510)
        at
com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:713)
        at
com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)
        at
com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3893)
        at
com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3231)
        at kafka.utils.Json$.encodeAsBytes(Json.scala:115)
        at kafka.zk.ReassignPartitionsZNode$.encode(ZkData.scala:403)
        at
kafka.zk.KafkaZkClient.createPartitionReassignment(KafkaZkClient.scala:843)
        at
kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:635)
        at
kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:221)
        at
kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:205)
        at
kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:65)
        at
kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)


Le 05.12.19 à 17:33, Ismael Juma a écrit :
> Are you using ZooKeeper 3.5.6 client libraries with Kafka 2.3.1? Kafka
> 2.3.1 ships with ZooKeeper 3.4.x.
>
> Ismael
>
> On Thu, Dec 5, 2019 at 8:18 AM Charles Bueche <cbueche@gmail.com> wrote:
>
>> Hello all,
>>
>> I do have two Kafka clusters in action, test and prod. The two are formed
>> by 3 nodes each, are independent and run their own zookeeper setups. My
>> prod cluster is running fine. My test cluster is half-broken and I'm
>> struggling to fix it. I could wipe it but I prefer to understand what's
>> wrong and fix it.
>>
>> I'm not sure what broke my test cluster. I had several network
>> disconnections / split-brains but Kafka always recovered fine. The reasons
>> for the network issues are independent and still being investigated
>> (layer-2 storms, etc).
>>
>> So I upgraded my zookeeper and kafka to the latest versions and when
>> trying to rebalance a topic across brokers I started to notice the
>> problems. Not sure really when they started, before or after the upgrade.
>>
>> I ran the upgrade as for the official doc (rolling upgrade, moving up
>> inter.broker.protocol.version and log.message.format.version gradually).
>>
>> ------------------------------------------------------
>> cat /etc/os-release
>> NAME="SLES"
>> VERSION="12-SP4"
>> VERSION_ID="12.4"
>> PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4"
>> ID="sles"
>> ANSI_COLOR="0;32"
>> CPE_NAME="cpe:/o:suse:sles:12:sp4"
>>
>> ------------------------------------------------------
>> versions :
>> - zookeeper 3.5.6
>> - kafka 2.12-2.3.1
>>
>> ------------------------------------------------------
>>
>> many rapid log entries on brokers 1 & 2 (we have 1, 2, 3)
>>
>> [2019-12-05 09:56:54,967] ERROR [KafkaApi-1] Number of alive brokers '0'
>> does not meet the required replication factor '1' for the offsets topic
>> (configured via 'offsets.topic.replication.factor'). This error can be
>> ignored if the cluster is starting up and not all brokers are up yet.
>> (kafka.server.KafkaApis)
>>
>> ------------------------------------------------------
>> java is java-1_8_0-openjdk-1.8.0.222-27.35.2.x86_64 from SLES. I have
>> tried Oracle Java jdk1.8.0_231 with the same issue.
>>
>> ------------------------------------------------------
>> when trying to see a reassignment I have this very suspect error :
>>
>> root@vmgato701a01:/appl/kafka/bin # ./kafka-reassign-partitions.sh
>> --zookeeper $ZKLIST --reassignment-json-file /tmp/r7.json --verify
>>
>> Status of partition reassignment:
>> Partitions reassignment failed due to
>> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
>> java.lang.AbstractMethodError:
>> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
>>         at
>> com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findDeserializer(OptionalHandlerFactory.java:122)
>>         at
>> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findOptionalStdDeserializer(BasicDeserializerFactory.java:1589)
>>         at
>> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1812)
>>         at
>> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:161)
>>         at
>> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:125)
>>         at
>> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411)
>>         at
>> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
>>         at
>> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
>>         at
>> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
>>         at
>> com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
>>         at
>> com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
>>         at
>> com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178)
>>         at
>> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
>>         at
>> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3079)
>>         at kafka.utils.Json$.parseBytesAs(Json.scala:73)
>>         at kafka.zk.ReassignPartitionsZNode$.decode(ZkData.scala:407)
>>         at
>> kafka.zk.KafkaZkClient.getPartitionReassignment(KafkaZkClient.scala:795)
>>         at
>> kafka.admin.ReassignPartitionsCommand$.checkIfPartitionReassignmentSucceeded(ReassignPartitionsCommand.scala:355)
>>         at
>> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:97)
>>         at
>> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90)
>>         at
>> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
>>         at
>> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
>>
>> Help appreciated.
>>
-- 
Charles Bueche <cblists@bueche.ch>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message