kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Bueche <cbli...@bueche.ch>
Subject kafka-reassign-partitions / JSON parsing broke between 2.1 and 2.2. Was : broken cluster / com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath
Date Thu, 12 Dec 2019 11:09:30 GMT
Hello again,

spending hours debugging this and having no clue...

  * Kafka 2.12-2.1.0 : kafka-reassign-partitions.sh is working fine
  * Anything newer (2.2.2, 2.3.1, trunk) breaks any operation involving
    JSON. Stack Dump below.

Help please.

Charles

Le 11.12.19 à 16:58, Charles Bueche a écrit :
> Hello again,
>
> Problem is still present, something is badly broken with the handling of
> JSON files. Details :
>
> I have made multiple downgrades and rolling-upgrades, and the situation
> is now :
>
>   * Zookeeper 3.4.14
>   * Kafka 2.12-2.3.1 (contains libs/zookeeper-3.4.14.jar)
>
> The cluster is all "happy", with balanced, no skew, etc. I have
> kafka-manager and everything is ok.
>
> I then create a "testing" topic :
>
>     export ZKLIST="srv1:2181,srv2:2181,srv3:2181"
>     /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --create
> --replication-factor 3 --partitions 1 --topic testing
>
> I can rise the partitions fine :
>
>     /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --alter --topic
> testing --partitions 3                          
>
> It then looks ok :
>
>     /appl/kafka/bin/kafka-topics.sh --zookeeper $ZKLIST --describe
> --topic testing                                      
>     Topic:testing   PartitionCount:3        ReplicationFactor:3     Configs:
>         Topic: testing  Partition: 0    Leader: 1       Replicas: 1,2,3
> Isr: 1,2,3
>         Topic: testing  Partition: 1    Leader: 2       Replicas: 2,3,1
> Isr: 2,3,1
>         Topic: testing  Partition: 2    Leader: 3       Replicas: 3,1,2
> Isr: 3,1,2
>
> Attempting any operation involving a json file breaks, when  it worked
> fine on Kafka 2.12-2.1.0. I shall precise my json files are passing lint
> tests and work ok.
>
> Helped needed !!!
>
> Thanks,
> Charles
>
> /appl/kafka/bin/kafka-reassign-partitions.sh --zookeeper $ZKLIST
> --reassignment-json-file ~buec/tmp/r7.json --execute
> Current partition replica assignment
>
> {"version":1,"partitions":[{"topic":"testing","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"testing","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"testing","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]}]}
>
> Save this to use as the --reassignment-json-file option during rollback
> Partitions reassignment failed due to
> com.fasterxml.jackson.databind.ext.Java7Support.getSerializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonSerializer;
> java.lang.AbstractMethodError:
> com.fasterxml.jackson.databind.ext.Java7Support.getSerializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonSerializer;
>         at
> com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findSerializer(OptionalHandlerFactory.java:92)
>         at
> com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findOptionalStdSerializer(BasicSerializerFactory.java:439)
>         at
> com.fasterxml.jackson.databind.ser.BasicSerializerFactory.findSerializerByPrimaryType(BasicSerializerFactory.java:374)
>         at
> com.fasterxml.jackson.databind.ser.BeanSerializerFactory._createSerializer2(BeanSerializerFactory.java:226)
>         at
> com.fasterxml.jackson.databind.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:165)
>         at
> com.fasterxml.jackson.databind.SerializerProvider._createUntypedSerializer(SerializerProvider.java:1385)
>         at
> com.fasterxml.jackson.databind.SerializerProvider._createAndCacheUntypedSerializer(SerializerProvider.java:1336)
>         at
> com.fasterxml.jackson.databind.SerializerProvider.findValueSerializer(SerializerProvider.java:510)
>         at
> com.fasterxml.jackson.databind.SerializerProvider.findTypedValueSerializer(SerializerProvider.java:713)
>         at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:308)
>         at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3893)
>         at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3231)
>         at kafka.utils.Json$.encodeAsBytes(Json.scala:115)
>         at kafka.zk.ReassignPartitionsZNode$.encode(ZkData.scala:403)
>         at
> kafka.zk.KafkaZkClient.createPartitionReassignment(KafkaZkClient.scala:843)
>         at
> kafka.admin.ReassignPartitionsCommand.reassignPartitions(ReassignPartitionsCommand.scala:635)
>         at
> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:221)
>         at
> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:205)
>         at
> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:65)
>         at
> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
>
>
> Le 05.12.19 à 17:33, Ismael Juma a écrit :
>> Are you using ZooKeeper 3.5.6 client libraries with Kafka 2.3.1? Kafka
>> 2.3.1 ships with ZooKeeper 3.4.x.
>>
>> Ismael
>>
>> On Thu, Dec 5, 2019 at 8:18 AM Charles Bueche <cbueche@gmail.com> wrote:
>>
>>> Hello all,
>>>
>>> I do have two Kafka clusters in action, test and prod. The two are formed
>>> by 3 nodes each, are independent and run their own zookeeper setups. My
>>> prod cluster is running fine. My test cluster is half-broken and I'm
>>> struggling to fix it. I could wipe it but I prefer to understand what's
>>> wrong and fix it.
>>>
>>> I'm not sure what broke my test cluster. I had several network
>>> disconnections / split-brains but Kafka always recovered fine. The reasons
>>> for the network issues are independent and still being investigated
>>> (layer-2 storms, etc).
>>>
>>> So I upgraded my zookeeper and kafka to the latest versions and when
>>> trying to rebalance a topic across brokers I started to notice the
>>> problems. Not sure really when they started, before or after the upgrade.
>>>
>>> I ran the upgrade as for the official doc (rolling upgrade, moving up
>>> inter.broker.protocol.version and log.message.format.version gradually).
>>>
>>> ------------------------------------------------------
>>> cat /etc/os-release
>>> NAME="SLES"
>>> VERSION="12-SP4"
>>> VERSION_ID="12.4"
>>> PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4"
>>> ID="sles"
>>> ANSI_COLOR="0;32"
>>> CPE_NAME="cpe:/o:suse:sles:12:sp4"
>>>
>>> ------------------------------------------------------
>>> versions :
>>> - zookeeper 3.5.6
>>> - kafka 2.12-2.3.1
>>>
>>> ------------------------------------------------------
>>>
>>> many rapid log entries on brokers 1 & 2 (we have 1, 2, 3)
>>>
>>> [2019-12-05 09:56:54,967] ERROR [KafkaApi-1] Number of alive brokers '0'
>>> does not meet the required replication factor '1' for the offsets topic
>>> (configured via 'offsets.topic.replication.factor'). This error can be
>>> ignored if the cluster is starting up and not all brokers are up yet.
>>> (kafka.server.KafkaApis)
>>>
>>> ------------------------------------------------------
>>> java is java-1_8_0-openjdk-1.8.0.222-27.35.2.x86_64 from SLES. I have
>>> tried Oracle Java jdk1.8.0_231 with the same issue.
>>>
>>> ------------------------------------------------------
>>> when trying to see a reassignment I have this very suspect error :
>>>
>>> root@vmgato701a01:/appl/kafka/bin # ./kafka-reassign-partitions.sh
>>> --zookeeper $ZKLIST --reassignment-json-file /tmp/r7.json --verify
>>>
>>> Status of partition reassignment:
>>> Partitions reassignment failed due to
>>> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
>>> java.lang.AbstractMethodError:
>>> com.fasterxml.jackson.databind.ext.Java7Support.getDeserializerForJavaNioFilePath(Ljava/lang/Class;)Lcom/fasterxml/jackson/databind/JsonDeserializer;
>>>         at
>>> com.fasterxml.jackson.databind.ext.OptionalHandlerFactory.findDeserializer(OptionalHandlerFactory.java:122)
>>>         at
>>> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findOptionalStdDeserializer(BasicDeserializerFactory.java:1589)
>>>         at
>>> com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1812)
>>>         at
>>> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:161)
>>>         at
>>> com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:125)
>>>         at
>>> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411)
>>>         at
>>> com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349)
>>>         at
>>> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264)
>>>         at
>>> com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244)
>>>         at
>>> com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142)
>>>         at
>>> com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:477)
>>>         at
>>> com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4178)
>>>         at
>>> com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3997)
>>>         at
>>> com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3079)
>>>         at kafka.utils.Json$.parseBytesAs(Json.scala:73)
>>>         at kafka.zk.ReassignPartitionsZNode$.decode(ZkData.scala:407)
>>>         at
>>> kafka.zk.KafkaZkClient.getPartitionReassignment(KafkaZkClient.scala:795)
>>>         at
>>> kafka.admin.ReassignPartitionsCommand$.checkIfPartitionReassignmentSucceeded(ReassignPartitionsCommand.scala:355)
>>>         at
>>> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:97)
>>>         at
>>> kafka.admin.ReassignPartitionsCommand$.verifyAssignment(ReassignPartitionsCommand.scala:90)
>>>         at
>>> kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:61)
>>>         at
>>> kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)
>>>
>>> Help appreciated.
>>>
-- 
Charles Bueche <cblists@bueche.ch>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message