kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shlomi Hazan <shl...@viber.com>
Subject Re: expanding cluster and reassigning parititions without restarting producer
Date Wed, 12 Nov 2014 06:04:57 GMT
Understood.
Thank you guys.

On Wed, Nov 12, 2014 at 4:48 AM, Jun Rao <junrao@gmail.com> wrote:

> Just to extend what Neha said. The new producer also picks up the new
> partitions by refreshing the metadata periodically (controlled
> metadata.max.age.ms). The new producer distributes the data more evenly to
> all partitions than the old producer.
>
> Thanks,
>
> Jun
>
> On Tue, Nov 11, 2014 at 11:19 AM, Neha Narkhede <neha.narkhede@gmail.com>
> wrote:
>
> > The new producer is available in 0.8.2-beta (the most recent Kafka
> > release). The old producer only detects new partitions at an interval
> > configured by topic.metadata.refresh.interval.ms. This constraint is no
> > longer true for the new producer and you would likely end up with an even
> > distribution of data across all partitions. If you want to stay with the
> > old producer on 0.8.1.1, you can try reducing
> > topic.metadata.refresh.interval.ms but it may have some performance
> impact
> > on the Kafka cluster since it ends up sending topic metadata requests to
> > the broker at that interval.
> >
> > Thanks,
> > Neha
> >
> > On Tue, Nov 11, 2014 at 1:45 AM, Shlomi Hazan <shlomi@viber.com> wrote:
> >
> > > Neha, I understand that the producer kafka.javaapi.producer.Producer
> > shown
> > > in examples is old,
> > > and that a new producer (org.apache.kafka.clients.producer) is avail?
> is
> > it
> > > available for 0.8.1.1?
> > > how does it work? does it have a trigger fired when partitions are
> added
> > or
> > > does the producer refresh some cache every some given time period?
> > >
> > > Shlomi
> > >
> > >
> > > On Tue, Nov 11, 2014 at 4:25 AM, Neha Narkhede <
> neha.narkhede@gmail.com>
> > > wrote:
> > >
> > > > How can I auto refresh keyed producers to use new partitions as these
> > > > partitions are added?
> > > >
> > > > Try using the new producer under org.apache.kafka.clients.producer.
> > > >
> > > > Thanks,
> > > > Neha
> > > >
> > > > On Mon, Nov 10, 2014 at 8:52 AM, Bhavesh Mistry <
> > > > mistry.p.bhavesh@gmail.com>
> > > > wrote:
> > > >
> > > > > I had different experience with expanding partition for new
> producer
> > > and
> > > > > its impact.  I only tried for non-key message.    I would always
> > advice
> > > > to
> > > > > keep batch size relatively low or plan for expansion with new java
> > > > producer
> > > > > in advance or since inception otherwise running producer code is
> > > > impacted.
> > > > >
> > > > > Here is mail chain:
> > > > >
> > > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201411.mbox/%3CCAOeJiJiT4CgRy97DGzfjKfVaQFDUv-O1x1KafEfbshGiRKM0Hw@mail.gmail.com%3E
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Bhavesh
> > > > >
> > > > > On Mon, Nov 10, 2014 at 5:20 AM, Shlomi Hazan <shlomi@viber.com>
> > > wrote:
> > > > >
> > > > > > Hmmm..
> > > > > > The Java producer example seems to ignore added partitions too...
> > > > > > How can I auto refresh keyed producers to use new partitions
as
> > these
> > > > > > partitions are added?
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 10, 2014 at 12:33 PM, Shlomi Hazan <shlomi@viber.com
> >
> > > > wrote:
> > > > > >
> > > > > > > One more thing:
> > > > > > > I saw that the Python client is also unaffected by addition
of
> > > > > partitions
> > > > > > > to a topic and that it continues to send requests only
to the
> old
> > > > > > > partitions.
> > > > > > > is this also handled appropriately by the Java producer?
Will
> he
> > > see
> > > > > the
> > > > > > > change and produce to the new partitions as well?
> > > > > > > Shlomi
> > > > > > >
> > > > > > > On Mon, Nov 10, 2014 at 9:34 AM, Shlomi Hazan <
> shlomi@viber.com>
> > > > > wrote:
> > > > > > >
> > > > > > >> No I don't see anything like that, the question was
aimed at
> > > > learning
> > > > > if
> > > > > > >> it is worthwhile to make the effort of reimplementing
the
> Python
> > > > > > producer
> > > > > > >> in Java, I so I will not make all the effort just to
be
> > > disappointed
> > > > > > >> afterwards.
> > > > > > >> understand I have nothing to worry about, so I will
try to
> > > simulate
> > > > > this
> > > > > > >> situation in small scale...
> > > > > > >> maybe 3 brokers, one topic with one partition and then
add
> > > > partitions.
> > > > > > >> we'll see.
> > > > > > >> thanks for clarifying.
> > > > > > >> Oh, Good luck with Confluent!!
> > > > > > >> :)
> > > > > > >>
> > > > > > >> On Mon, Nov 10, 2014 at 4:17 AM, Neha Narkhede <
> > > > > neha.narkhede@gmail.com
> > > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> The producer might get an error code if the leader
of the
> > > > partitions
> > > > > > >>> being
> > > > > > >>> reassigned also changes. However it should retry
and succeed.
> > Do
> > > > you
> > > > > > see
> > > > > > >>> a
> > > > > > >>> behavior that suggests otherwise?
> > > > > > >>>
> > > > > > >>> On Sat, Nov 8, 2014 at 11:45 PM, Shlomi Hazan <
> > shlomi@viber.com>
> > > > > > wrote:
> > > > > > >>>
> > > > > > >>> > Hi All,
> > > > > > >>> > I recently had an issue producing from python
where
> > expanding a
> > > > > > cluster
> > > > > > >>> > from 3 to 5 nodes and reassigning partitions
forced me to
> > > restart
> > > > > the
> > > > > > >>> > producer b/c of KeyError thrown.
> > > > > > >>> > Is this situation handled by the Java producer
> automatically
> > or
> > > > > need
> > > > > > I
> > > > > > >>> do
> > > > > > >>> > something to have the java producer refresh
itself to see
> the
> > > > > > >>> reassigned
> > > > > > >>> > partition layout and produce away ?
> > > > > > >>> > Shlomi
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message