kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Peterson <d...@dspeterson.com>
Subject Re: trouble upgrading from 0.8.2.1 to 0.9.0.0: invalid message
Date Tue, 19 Jan 2016 16:08:57 GMT
I'll try again and look for logged errors.  The problem I saw with
kafka-console-producer.sh did occur with a mixed version cluster,
so that is likely the problem.  I'll try again with the 0.8.2.1 version.

Thanks,
Dave


On 1/19/2016 2:50 AM, Ismael Juma wrote:
> Hi Dave,
>
> Do you get any errors logged in the broker when you get ACK error 2
> (InvalidMessage) while producing requests to a mixed version cluster? It
> would be helpful to see them.
>
> With regards to the kafka-console-producer.sh error, did you use the
> 0.9.0.0 console producer with a mixed version cluster (ie some brokers were
> on 0.8.2.1 while others were on 0.9.0.0)? If so, it is expected that it
> won't work correctly. All the brokers should be upgraded before the clients
> are upgraded (otherwise the 0.8.2.1 broker will send a response that the
> newer clients cannot handle).
>
> Ismael
>
> On Fri, Jan 15, 2016 at 7:52 PM, Dave Peterson <dave@academia.edu> wrote:
>
>> Hi Ismael,
>>
>> I'm using bruce (https://github.com/ifwe/bruce) to send the produce
>> requests, with a RequiredAcks value of 1.  Everything works fine when
>> all brokers are running 0.8.2.1.  Also if I set up a new 0.9.0.0
>> cluster from scratch rather than trying to upgrade, everything works
>> fine.  The problem only occurs after upgrading one broker in the
>> 3-broker cluster.
>>
>> The topic I am sending to has 8 partitions numbered 0-7.  Doing
>> further experimentation I see that the ACK error 2 occurs only when
>> I send to partition 7.  No problems occur when sending to partitions
>> 0-6.  If it helps I can send output from "kafka-topics.sh --describe"
>> as well as tcpdump output showing the produce requests and responses.
>>
>> For comparison I tried using the 0.9.0.0 version of
>> kafka-console-producer.sh to send messages.  With the default
>> RequiredAcks value of 0, it worked although I don't know which
>> partition it sent to.  With a RequiredAcks value of 1 I get the
>> output shown below.
>>
>> Thanks,
>> Dave
>>
>>
>>
>> [2016-01-15 10:33:28,843] ERROR Uncaught error in kafka producer I/O
>> thread:  (org.apache.kafka.clients.producer.internals.Sender)
>> org.apache.kafka.common.protocol.types.SchemaException: Error reading field
>> 'throttle_time_ms': java.nio.BufferUnderflowException
>>          at
>> org.apache.kafka.common.protocol.types.Schema.read(Schema.java:71)
>>          at
>>
>> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:464)
>>          at
>> org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:279)
>>          at
>> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
>>          at
>> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
>>          at java.lang.Thread.run(Thread.java:745)
>>
>>
>> On Fri, Jan 15, 2016 at 1:06 AM, Ismael Juma <ismael@juma.me.uk> wrote:
>>
>>> Hi Dave,
>>>
>>> On Fri, Jan 15, 2016 at 2:04 AM, Dave Peterson <dave@academia.edu>
>> wrote:
>>>> I was trying to upgrade an 0.8.2.1 broker cluster to 0.9.0.0 by
>> following
>>>> the instructions here:
>>>>
>>>>      http://kafka.apache.org/documentation.html#upgrade
>>>>
>>>> After upgrading one broker, with inter.broker.protocol.version=0.8.2.X
>>>> set, I get ACK error 2 (InvalidMessage) when I try to send produce
>>>> requests.
>>>
>>> I haven't seen other reports of this issue yet. Also, we have a system
>> test
>>> that covers this scenario:
>>>
>>>
>>>
>> https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/upgrade_test.py
>>> Just to double-check, what is the version of the producer that you are
>>> using to send produce requests to the 0.9.0.0 broker when you get the
>>> error?
>>>
>>> Ismael
>>>


Mime
View raw message