kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Drew Goya <d...@gradientx.com>
Subject Re: Producer not distributing across all partitions
Date Fri, 13 Sep 2013 20:17:14 GMT
I ran into this problem as well Prashant.  The default partition key was
recently changed:

https://github.com/apache/kafka/commit/b71e6dc352770f22daec0c9a3682138666f032be

It no longer assigns a random partition to data with a null partition key.
 I had to change my code to generate random partition keys to get the
randomly distributed behavior the producer used to have.


On Fri, Sep 13, 2013 at 11:42 AM, prashant amar <amasindhu@gmail.com> wrote:

> Thanks Neha
>
> I will try applying this property and circle back.
>
> Also, I have been attempting to execute kafka-producer-perf-test.sh and I
> receive the following error
>
>        Error: Could not find or load main class
> kafka.perf.ProducerPerformance
>
> I am running against 0.8.0-beta1
>
> Seems like perf is a separate project in the workspace.
>
> Does sbt package-assembly bundle the perf jar as well?
>
> Neither producer-perf-test not consumer-test are working with this build
>
>
>
> On Fri, Sep 13, 2013 at 9:56 AM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > As Jun suggested, one reason could be that the
> > topic.metadata.refresh.interval.ms is too high. Did you observe if the
> > distribution improves after topic.metadata.refresh.interval.ms has
> passed
> > ?
> >
> > Thanks
> > Neha
> >
> >
> > On Fri, Sep 13, 2013 at 4:47 AM, prashant amar <amasindhu@gmail.com>
> > wrote:
> >
> > > I am using kafka 08 version ...
> > >
> > >
> > > On Thu, Sep 12, 2013 at 8:44 PM, Jun Rao <junrao@gmail.com> wrote:
> > >
> > > > Which revision of 0.8 are you using? In a recent change, a producer
> > will
> > > > stick to a partition for topic.metadata.refresh.interval.ms(defaults
> > to
> > > > 10
> > > > mins) time before picking another partition at random.
> > > > Thanks,
> > > > Jun
> > > >
> > > >
> > > > On Thu, Sep 12, 2013 at 1:56 PM, prashant amar <amasindhu@gmail.com>
> > > > wrote:
> > > >
> > > > > I created a topic with 4 partitions and for some reason the
> producer
> > is
> > > > > pushing only to one partition.
> > > > >
> > > > > This is consistently happening across all topics that I created ...
> > > > >
> > > > > Is there a specific configuration that I need to apply to ensure
> that
> > > > load
> > > > > is evenly distributed across all partitions?
> > > > >
> > > > >
> > > > > Group           Topic                          Pid Offset
> > > >  logSize
> > > > >         Lag             Owner
> > > > > perfgroup1      perfpayload1                   0   10965
> > > 11220
> > > > >         255             perfgroup1_XXXX-0
> > > > > perfgroup1      perfpayload1                   1   0
> 0
> > > > >         0               perfgroup1_XXXX-1
> > > > > perfgroup1      perfpayload1                   2   0
> 0
> > > > >         0               perfgroup1_XXXXX-2
> > > > > perfgroup1      perfpayload1                   3   0
> 0
> > > > >         0               perfgroup1_XXXXX-3
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message