kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <Manoj.Agraw...@cognizant.com>
Subject Re: backup
Date Sun, 09 Aug 2020 18:04:23 GMT
I'm also working on mirror maker 2.0 . Do you have any documentation for mirror maker 2.0 config
setup or can you share the mirror maker 2.0 config .
Have you encounter any issue

´╗┐On 8/9/20, 5:02 AM, "Liam Clarke-Hutchinson" <liam.clarke@adscale.co.nz> wrote:

    [External]


    Hi Dor,

    Yep, we're using Mirrormaker 2.0 currently, migrated from MM 1 with no real
    issues. Admittedly the documentation is a bit lacking currently, but
    between the KIP
    <https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-382%253A%2BMirrorMaker%2B2.0&amp;data=02%7C01%7CManoj.Agrawal2%40cognizant.com%7Cf0d6e338899a47c5eb5a08d83c5c1023%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637325713403576927&amp;sdata=kjSo0G9ntnSVKgfnLf4niPUy8nQWx%2Bsw5x14UKhTIxE%3D&amp;reserved=0>
    and
    the occasional code reading, we got there fine. One caveat to my experience
    - it's built on top of Kafka Connect, and as we already had a cluster of KC
    workers to stream data into/out of our Kafka cluster from various sources,
    MM 2 was easy to deploy, just another config. So if you're starting from
    scratch, there might be some overhead around getting your Kafka Connect
    workers deployed and configured correctly. But if you're looking to run a
    hot/warm (main/backup) cluster scenario, MM 2 is ideal.

    Kind regards,

    Liam Clarke-Hutchinson

    On Sun, Aug 9, 2020 at 11:56 PM Dor Ben Dov <dor.ben-dov@amdocs.com> wrote:

    > Hi Liam,
    > No actual problem just wondering, still you answered most of the things I
    > already know so no I am convinced that I am ok.
    > Still, wondering about the mmk2. How reliable is it, have you used it in
    > production for instance?
    >
    > Regards,
    > Dor
    >
    > -----Original Message-----
    > From: Liam Clarke-Hutchinson <liam.clarke@adscale.co.nz>
    > Sent: Sunday, August 9, 2020 2:52 PM
    > To: users@kafka.apache.org
    > Subject: Re: backup
    >
    > Hi Dor,
    >
    > There are multiple approaches.
    >
    > 1) Clone your Kafka broker volumes
    > 2) Use Kafka Connect to stream all data to a different storage system such
    > as Hadoop, S3, etc.
    > 3) Use Mirrormaker to replicate all data to a backup cluster.
    >
    > Which approach is right for you really depends on your needs, but
    > generally, if you have enough nodes in your clusters, and a correct
    > replication setting for a topic, you won't need to backup Kafka. As a rule
    > of thumb, a topic with a replication factor of N can survive N - 1 node
    > failures without data loss.
    >
    > If you can provide more information about the problems you're trying to
    > solve, our advice can be more directed :)
    >
    > Kind regards,
    >
    > Liam Clarke-Hutchinson
    >
    > On Sun, Aug 9, 2020 at 11:43 PM Dor Ben Dov <dor.ben-dov@amdocs.com>
    > wrote:
    >
    > > Hi All,
    > > What is the best recommended way, and tool to backup kafka in production?
    > > Regards,
    > > Dor
    > >
    > > This email and the information contained herein is proprietary and
    > > confidential and subject to the Amdocs Email Terms of Service, which
    > > you may review at
    > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
    > > amdocs.com%2Fabout%2Femail-terms-of-service&amp;data=02%7C01%7Cdor.ben
    > > -dov%40amdocs.com%7C41b5e7ba2ac947d8f92208d83c5ac7c3%7Cc8eca3ca127646d
    > > 59d9da0f2a028920f%7C0%7C0%7C637325707889579052&amp;sdata=jXXwnZCqI5bjQ
    > > 8beZTR8WM7l1yhEPkuSQJ%2FMqfJlz00%3D&amp;reserved=0 <
    > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
    > > amdocs.com%2Fabout%2Femail-terms-of-service&amp;data=02%7C01%7Cdor.ben
    > > -dov%40amdocs.com%7C41b5e7ba2ac947d8f92208d83c5ac7c3%7Cc8eca3ca127646d
    > > 59d9da0f2a028920f%7C0%7C0%7C637325707889579052&amp;sdata=jXXwnZCqI5bjQ
    > > 8beZTR8WM7l1yhEPkuSQJ%2FMqfJlz00%3D&amp;reserved=0>
    > >
    > >
    > This email and the information contained herein is proprietary and
    > confidential and subject to the Amdocs Email Terms of Service, which you
    > may review at https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amdocs.com%2Fabout%2Femail-terms-of-service&amp;data=02%7C01%7CManoj.Agrawal2%40cognizant.com%7Cf0d6e338899a47c5eb5a08d83c5c1023%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637325713403576927&amp;sdata=GR9B57ECse8QETY8nAjmPn7w12n5ezMCj12%2FmP1BQd4%3D&amp;reserved=0
<
    > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amdocs.com%2Fabout%2Femail-terms-of-service&amp;data=02%7C01%7CManoj.Agrawal2%40cognizant.com%7Cf0d6e338899a47c5eb5a08d83c5c1023%7Cde08c40719b9427d9fe8edf254300ca7%7C0%7C0%7C637325713403586918&amp;sdata=Q%2BARIp1ASnZdXhtkCGot8gjs2%2BeW%2BVjmsPek7g%2FUPqw%3D&amp;reserved=0>
    >
    >


This e-mail and any files transmitted with it are for the sole use of the intended recipient(s)
and may contain confidential and privileged information. If you are not the intended recipient(s),
please reply to the sender and destroy all copies of the original message. Any unauthorized
review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or
any action taken in reliance on the contents of this e-mail is strictly prohibited and may
be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications
sent to and from Cognizant e-mail addresses may be monitored.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s)
and may contain confidential and privileged information. If you are not the intended recipient(s),
please reply to the sender and destroy all copies of the original message. Any unauthorized
review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or
any action taken in reliance on the contents of this e-mail is strictly prohibited and may
be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications
sent to and from Cognizant e-mail addresses may be monitored.
Mime
View raw message