kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishal Santoshi <vishal.santo...@gmail.com>
Subject Re: MirrorMaker 2.0 ( working off origin/2.4 )
Date Mon, 04 Nov 2019 20:26:41 GMT
I can verify that the above did take. ( kicking myself ) . It should be the
same for, these too ?

b.producer.batch.size = 1048576

b.producer.linger.ms  = 30

b.producer.acks = 1


etc etc...


I also see that the properties can be overridden, so this routine


* kill 1 MM2

* change the mm2.properites

* restart mm2


should work provided we do not change the client_id


Thanks.



On Mon, Nov 4, 2019 at 3:08 PM Ryanne Dolan <ryannedolan@gmail.com> wrote:

> > BTW any ideas when 2.4 is being released
>
> Looks like there are a few blockers still.
>
> On Mon, Nov 4, 2019 at 2:06 PM Vishal Santoshi <vishal.santoshi@gmail.com>
> wrote:
>
> > I bet I have tested the "b.producer.acks' route. I will test again and
> let
> > you know. Note that I resorted to hardcoding that value in the Sender and
> > that alleviated the throttle I was seeing on consumption.  BTW any ideas
> > when 2.4 is being released ( I thought it was Oct 30th 2019 )...
> >
> > On Mon, Nov 4, 2019 at 1:49 PM Ryanne Dolan <ryannedolan@gmail.com>
> wrote:
> >
> > > Vishal, b.producer.acks should work, as can be seen in the following
> unit
> > > test with similar producer property "client.id":
> > >
> > >
> > >
> >
> https://github.com/apache/kafka/blob/6b905ade0cdc7a5f6f746727ecfe4e7a7463a200/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/MirrorMakerConfigTest.java#L182
> > >
> > > Keep in mind this should be configured for the target cluster, and it
> > might
> > > not take effect until the workers finish rebalancing.
> > >
> > > Ryanne
> > >
> > > On Sun, Nov 3, 2019, 9:40 AM Vishal Santoshi <
> vishal.santoshi@gmail.com>
> > > wrote:
> > >
> > > > Hello folks,
> > > >                   Was doing stress tests and realized that the
> > > replication
> > > > to the target cluster and thus the configuration of the KafkaProducer
> > > has a
> > > > default acks of -1 ( all ) and that was prohibitively expensive. It
> > > should
> > > > have been a simple a->b.producer.acks = 1 ( or  b.producer.acks = 1
> or
> > > acks
> > > > = 1, have tried all ) override in the mm2.properties, but it does not
> > > take.
> > > > I verified by turning on TRACE and a little  instrumenting the code
> to
> > > > print out the acks .
> > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L806
> > > > .
> > > >  Am I missing something?
> > > >
> > > > Regards
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message