kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shlomi Hazan <shl...@viber.com>
Subject Re: expanding cluster and reassigning parititions without restarting producer
Date Tue, 11 Nov 2014 09:45:41 GMT
Neha, I understand that the producer kafka.javaapi.producer.Producer shown
in examples is old,
and that a new producer (org.apache.kafka.clients.producer) is avail? is it
available for 0.8.1.1?
how does it work? does it have a trigger fired when partitions are added or
does the producer refresh some cache every some given time period?

Shlomi


On Tue, Nov 11, 2014 at 4:25 AM, Neha Narkhede <neha.narkhede@gmail.com>
wrote:

> How can I auto refresh keyed producers to use new partitions as these
> partitions are added?
>
> Try using the new producer under org.apache.kafka.clients.producer.
>
> Thanks,
> Neha
>
> On Mon, Nov 10, 2014 at 8:52 AM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com>
> wrote:
>
> > I had different experience with expanding partition for new producer and
> > its impact.  I only tried for non-key message.    I would always advice
> to
> > keep batch size relatively low or plan for expansion with new java
> producer
> > in advance or since inception otherwise running producer code is
> impacted.
> >
> > Here is mail chain:
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201411.mbox/%3CCAOeJiJiT4CgRy97DGzfjKfVaQFDUv-O1x1KafEfbshGiRKM0Hw@mail.gmail.com%3E
> >
> > Thanks,
> >
> > Bhavesh
> >
> > On Mon, Nov 10, 2014 at 5:20 AM, Shlomi Hazan <shlomi@viber.com> wrote:
> >
> > > Hmmm..
> > > The Java producer example seems to ignore added partitions too...
> > > How can I auto refresh keyed producers to use new partitions as these
> > > partitions are added?
> > >
> > >
> > > On Mon, Nov 10, 2014 at 12:33 PM, Shlomi Hazan <shlomi@viber.com>
> wrote:
> > >
> > > > One more thing:
> > > > I saw that the Python client is also unaffected by addition of
> > partitions
> > > > to a topic and that it continues to send requests only to the old
> > > > partitions.
> > > > is this also handled appropriately by the Java producer? Will he see
> > the
> > > > change and produce to the new partitions as well?
> > > > Shlomi
> > > >
> > > > On Mon, Nov 10, 2014 at 9:34 AM, Shlomi Hazan <shlomi@viber.com>
> > wrote:
> > > >
> > > >> No I don't see anything like that, the question was aimed at
> learning
> > if
> > > >> it is worthwhile to make the effort of reimplementing the Python
> > > producer
> > > >> in Java, I so I will not make all the effort just to be disappointed
> > > >> afterwards.
> > > >> understand I have nothing to worry about, so I will try to simulate
> > this
> > > >> situation in small scale...
> > > >> maybe 3 brokers, one topic with one partition and then add
> partitions.
> > > >> we'll see.
> > > >> thanks for clarifying.
> > > >> Oh, Good luck with Confluent!!
> > > >> :)
> > > >>
> > > >> On Mon, Nov 10, 2014 at 4:17 AM, Neha Narkhede <
> > neha.narkhede@gmail.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> The producer might get an error code if the leader of the
> partitions
> > > >>> being
> > > >>> reassigned also changes. However it should retry and succeed.
Do
> you
> > > see
> > > >>> a
> > > >>> behavior that suggests otherwise?
> > > >>>
> > > >>> On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan <shlomi@viber.com>
> > > wrote:
> > > >>>
> > > >>> > Hi All,
> > > >>> > I recently had an issue producing from python where expanding
a
> > > cluster
> > > >>> > from 3 to 5 nodes and reassigning partitions forced me to
restart
> > the
> > > >>> > producer b/c of KeyError thrown.
> > > >>> > Is this situation handled by the Java producer automatically
or
> > need
> > > I
> > > >>> do
> > > >>> > something to have the java producer refresh itself to see
the
> > > >>> reassigned
> > > >>> > partition layout and produce away ?
> > > >>> > Shlomi
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message