kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörg Wagner <joerg.wagn...@1und1.de>
Subject Re: Experiences with corrupted messages
Date Fri, 02 Oct 2015 10:49:13 GMT
Thank you very much.

I will check that, but it does seem to stop on the same message.

Jörg

On 01.10.2015 22:49, Lance Laursen wrote:
> Hey Jörg,
>
> Unfortunately when the high level consumer hits a corrupt message, it
> enters an invalid state and closes. The only way around this is to iterate
> your offset by 1 in order to skip the corrupt message. This is currently
> not automated. You can catch this exception if you are using the simple
> consumer client, but unfortunately mirrormaker uses the high level client.
>
> There have been some corrupt producer message bugs related to using snappy
> compression recently, but this does not seem to be the same as your problem.
>
> Does MM stop on the exact same message each time (
> https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-ConsumerOffsetChecker
> )? I would suggest triple checking that your configurations are the same
> across all DC's (you mentioned that MM mirrors successfully to another DC
> with no problem), as well as examine the problem message to see if you can
> find anything different about it when compared to the others (See:
> https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-SimpleConsumerShell
> ). Your only other recourse is to iterate past the problem offset.
>
> On Thu, Oct 1, 2015 at 1:22 AM, Jörg Wagner <joerg.wagner1@1und1.de> wrote:
>
>> Hey everyone,
>>
>> I've been having some issues with corrupted messages and mirrormaker as I
>> wrote previously. Since there was no feedback, I want to ask a new question:
>>
>> Did you ever have corrupted messages in kafka? Did things break? How did
>> you recover or work around that?
>>
>> Thanks
>> Jörg
>>

-- 
Mit freundlichem Gruß

Jörg Wagner

Systemadministrator
Search & Account Security

1&1 Mail & Media Development  & Technology GmbH | Sapporobogen 6-8 | 80637 München
| Germany
Phone: +49 89 14339 324
E-Mail: joerg.wagner1@1und1.de | Web: www.1und1.de

Amtsgericht Montabaur, HRB 5452

Geschäftsführer: Frank Einhellinger, Hans-Henning Kettler, Jan Oetjen


Member of United Internet


Mime
View raw message