kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vishal Santoshi <vishal.santo...@gmail.com>
Subject Re: MM2 startup delay
Date Mon, 23 Dec 2019 03:35:02 GMT
And can you share the patch...

On Sun, Dec 22, 2019 at 10:34 PM Vishal Santoshi <vishal.santoshi@gmail.com>
wrote:

> We also have a large number of topics 1500 plus and in a cross DC
> replication. How do we increase the default timeouts ?
>
>
> On Wed, Dec 11, 2019 at 2:26 PM Ryanne Dolan <ryannedolan@gmail.com>
> wrote:
>
>> Hey Peter. Do you see any timeouts in the logs? The internal scheduler
>> will
>> timeout each task after 60 seconds by default, which might not be long
>> enough to finish some of the bootstrap tasks in your case. My team has
>> observed that behavior in super-flaky environments, e.g. when connectivity
>> drops during bootstrapping, in which case MirrorSourceConnector can get
>> into a funky state. This resolves when it refreshes its state after a
>> while. The default refresh interval of 10 minutes seems to jibe with your
>> observations.
>>
>> My team patched our internal MM2 build months ago to force bootstrapping
>> to
>> complete correctly. I can share the patch, and if it helps we can raise a
>> PR.
>>
>> Ryanne
>>
>> On Mon, Dec 9, 2019 at 5:28 AM Péter Sinóros-Szabó
>> <peter.sinoros-szabo@transferwise.com.invalid> wrote:
>>
>> > Hi,
>> >
>> > I am experimenting with Mirror Make 2 in 2.4.0-rc3. It seems to start up
>> > fine, connects to both source and destination, creates new topics...
>> > But it does not start to actually mirror the messages until about 12
>> > minutes after MM2 was started. I would expect it to start mirroring in
>> some
>> > seconds after startup.
>> >
>> > Source cluster has about 2800 partitions, destination cluster is empty.
>> > Both clusters are in AWS but in different regions.
>> >
>> > What may cause the 12 minutes delay?
>> >
>> > Config is:
>> > ---
>> > clusters = eucmain, euwbackup
>> > eucmain.bootstrap.servers =
>> > test-kafka-main-fra01.xx:9092,test-kafka-main-fra02.xx:9092
>> > euwbackup.bootstrap.servers = 172.30.197.203:9092,172.30.213.104:9092
>> > eucmain->euwbackup.enabled = true
>> > eucmain->euwbackup.topics = .*
>> > eucmain->euwbackup.topics.blacklist = ^(kafka|kmf|__).*
>> > eucmain->euwbackup.rename.topics = false
>> > replication.policy.separator = __
>> > eucmain.client.id = mm2
>> >
>> > I do not see any serious errors in the logs that I would think of a
>> cause
>> > of this.
>> >
>> > Thanks,
>> > Peter
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message