kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Asaf Mesika <asaf.mes...@gmail.com>
Subject Re: Partition size skew using default partitioner (without key)
Date Tue, 05 Jul 2016 05:38:40 GMT
Apparently it's documented in the FAQ - but I ignored it since it said
"0.8.0" and I was using 0.8.2.1. After reading all the lengthy forum post
dating to 2013: The problematic code there is in DefaultEventHandler.scala,
but if I'm only using KafkaProducer.java - the java flavor - I won't be
exposed to this behaviour since you moved to NIO thus have a single socket
per broker?

On Tue, Jul 5, 2016 at 7:14 AM Asaf Mesika <asaf.mesika@gmail.com> wrote:

> Since the image is now shown, here's a direct link to it:
> https://s32.postimg.org/xoet3vu2t/image.png
>
> On Tue, Jul 5, 2016 at 7:01 AM Asaf Mesika <asaf.mesika@gmail.com> wrote:
>
>> As we continue to track down the cause I'm trying to ping back here in
>> case someone new might have an answer to the question below?
>>
>>
>> On Thu, Jun 16, 2016 at 12:39 PM Asaf Mesika <asaf.mesika@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> We've noticed that we have some partitions receiving more messages than
>>> others. What I've done to learn that is:
>>> * In Kafka Manager, per a given topic, the list of Partition Information
>>> is displayed.
>>> * For each partition there's a column called Latest Offset - which I
>>> assume is the producer offset. As I understand, this is the total number of
>>> messages written to this partition since the topic was created (and/or the
>>> partition was added)
>>> * I plotted the two columns: Partition number and Latest Offset.
>>>
>>> This is what I got:
>>> [image: pasted1]
>>>
>>> I'm using Kafka 0.8.2.1 with the new Producer API. We've using default
>>> partitioner, and we're *not* supplying partition number nor a key, thus
>>> it should be round-robin.
>>>
>>> From some reason, we see this.
>>>
>>> I was wondering if someone from the community ever encountered such a
>>> behaviour?
>>>
>>> Thanks!
>>>
>>> Asaf Mesika
>>> Logz.io
>>>
>>>

Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message