kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: expanding cluster and reassigning parititions without restarting producer
Date Wed, 12 Nov 2014 02:48:10 GMT
Just to extend what Neha said. The new producer also picks up the new
partitions by refreshing the metadata periodically (controlled
metadata.max.age.ms). The new producer distributes the data more evenly to
all partitions than the old producer.

Thanks,

Jun

On Tue, Nov 11, 2014 at 11:19 AM, Neha Narkhede <neha.narkhede@gmail.com>
wrote:

> The new producer is available in 0.8.2-beta (the most recent Kafka
> release). The old producer only detects new partitions at an interval
> configured by topic.metadata.refresh.interval.ms. This constraint is no
> longer true for the new producer and you would likely end up with an even
> distribution of data across all partitions. If you want to stay with the
> old producer on 0.8.1.1, you can try reducing
> topic.metadata.refresh.interval.ms but it may have some performance impact
> on the Kafka cluster since it ends up sending topic metadata requests to
> the broker at that interval.
>
> Thanks,
> Neha
>
> On Tue, Nov 11, 2014 at 1:45 AM, Shlomi Hazan <shlomi@viber.com> wrote:
>
> > Neha, I understand that the producer kafka.javaapi.producer.Producer
> shown
> > in examples is old,
> > and that a new producer (org.apache.kafka.clients.producer) is avail? is
> it
> > available for 0.8.1.1?
> > how does it work? does it have a trigger fired when partitions are added
> or
> > does the producer refresh some cache every some given time period?
> >
> > Shlomi
> >
> >
> > On Tue, Nov 11, 2014 at 4:25 AM, Neha Narkhede <neha.narkhede@gmail.com>
> > wrote:
> >
> > > How can I auto refresh keyed producers to use new partitions as these
> > > partitions are added?
> > >
> > > Try using the new producer under org.apache.kafka.clients.producer.
> > >
> > > Thanks,
> > > Neha
> > >
> > > On Mon, Nov 10, 2014 at 8:52 AM, Bhavesh Mistry <
> > > mistry.p.bhavesh@gmail.com>
> > > wrote:
> > >
> > > > I had different experience with expanding partition for new producer
> > and
> > > > its impact.  I only tried for non-key message.    I would always
> advice
> > > to
> > > > keep batch size relatively low or plan for expansion with new java
> > > producer
> > > > in advance or since inception otherwise running producer code is
> > > impacted.
> > > >
> > > > Here is mail chain:
> > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201411.mbox/%3CCAOeJiJiT4CgRy97DGzfjKfVaQFDUv-O1x1KafEfbshGiRKM0Hw@mail.gmail.com%3E
> > > >
> > > > Thanks,
> > > >
> > > > Bhavesh
> > > >
> > > > On Mon, Nov 10, 2014 at 5:20 AM, Shlomi Hazan <shlomi@viber.com>
> > wrote:
> > > >
> > > > > Hmmm..
> > > > > The Java producer example seems to ignore added partitions too...
> > > > > How can I auto refresh keyed producers to use new partitions as
> these
> > > > > partitions are added?
> > > > >
> > > > >
> > > > > On Mon, Nov 10, 2014 at 12:33 PM, Shlomi Hazan <shlomi@viber.com>
> > > wrote:
> > > > >
> > > > > > One more thing:
> > > > > > I saw that the Python client is also unaffected by addition
of
> > > > partitions
> > > > > > to a topic and that it continues to send requests only to the
old
> > > > > > partitions.
> > > > > > is this also handled appropriately by the Java producer? Will
he
> > see
> > > > the
> > > > > > change and produce to the new partitions as well?
> > > > > > Shlomi
> > > > > >
> > > > > > On Mon, Nov 10, 2014 at 9:34 AM, Shlomi Hazan <shlomi@viber.com>
> > > > wrote:
> > > > > >
> > > > > >> No I don't see anything like that, the question was aimed
at
> > > learning
> > > > if
> > > > > >> it is worthwhile to make the effort of reimplementing the
Python
> > > > > producer
> > > > > >> in Java, I so I will not make all the effort just to be
> > disappointed
> > > > > >> afterwards.
> > > > > >> understand I have nothing to worry about, so I will try
to
> > simulate
> > > > this
> > > > > >> situation in small scale...
> > > > > >> maybe 3 brokers, one topic with one partition and then add
> > > partitions.
> > > > > >> we'll see.
> > > > > >> thanks for clarifying.
> > > > > >> Oh, Good luck with Confluent!!
> > > > > >> :)
> > > > > >>
> > > > > >> On Mon, Nov 10, 2014 at 4:17 AM, Neha Narkhede <
> > > > neha.narkhede@gmail.com
> > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >>> The producer might get an error code if the leader of
the
> > > partitions
> > > > > >>> being
> > > > > >>> reassigned also changes. However it should retry and
succeed.
> Do
> > > you
> > > > > see
> > > > > >>> a
> > > > > >>> behavior that suggests otherwise?
> > > > > >>>
> > > > > >>> On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan <
> shlomi@viber.com>
> > > > > wrote:
> > > > > >>>
> > > > > >>> > Hi All,
> > > > > >>> > I recently had an issue producing from python where
> expanding a
> > > > > cluster
> > > > > >>> > from 3 to 5 nodes and reassigning partitions forced
me to
> > restart
> > > > the
> > > > > >>> > producer b/c of KeyError thrown.
> > > > > >>> > Is this situation handled by the Java producer
automatically
> or
> > > > need
> > > > > I
> > > > > >>> do
> > > > > >>> > something to have the java producer refresh itself
to see the
> > > > > >>> reassigned
> > > > > >>> > partition layout and produce away ?
> > > > > >>> > Shlomi
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message