kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liam Clarke-Hutchinson <liam.cla...@adscale.co.nz>
Subject Re: backup
Date Mon, 10 Aug 2020 12:22:46 GMT
Hi Kumar,

You can restore but you'll lose data since your last snapshot. I use KC ->
SO instead for obvious reasons.

All that said, I've never had multiple Kafka nodes fail to the extent that
I need my S3 backup. But it's good to have it, for peace of mind.

Thanks,

Liam Clarke-Hutchinson

On Mon, 10 Aug. 2020, 2:27 pm kumar, <oraclept@gmail.com> wrote:

> Hi Liam -  I did not understand cloning kafka broker volumes.
>
> if  you have 1 TB disk , Assuming the usage is 65%  data in the volume is
> changing so fast.
> take 650 GB every hour or every min ? how do we restore if there was
> failure?
>
> Oracle database provides point in time recovery(incremental+full
> backup+archive logs) of their database. is it possible to recover kafka
> like that?
> We had storage failure of the entire site. We were not confident on the
> data recovered on kafka compared to oracle database. We had 3 kafka nodes
> with no mirror maker.
>
> My understanding is  replication and mirror maker works until there is no
> lag on replication.  There is no guarantee of data loss.
> I never tested mirror maker with compact topics.
>
> Thanks,
> pt
>
>
>
>
> On Sun, Aug 9, 2020 at 7:52 AM Liam Clarke-Hutchinson <
> liam.clarke@adscale.co.nz> wrote:
>
> > Hi Dor,
> >
> > There are multiple approaches.
> >
> > 1) Clone your Kafka broker volumes
> > 2) Use Kafka Connect to stream all data to a different storage system
> such
> > as Hadoop, S3, etc.
> > 3) Use Mirrormaker to replicate all data to a backup cluster.
> >
> > Which approach is right for you really depends on your needs, but
> > generally, if you have enough nodes in your clusters, and a correct
> > replication setting for a topic, you won't need to backup Kafka. As a
> rule
> > of thumb, a topic with a replication factor of N can survive N - 1 node
> > failures without data loss.
> >
> > If you can provide more information about the problems you're trying to
> > solve, our advice can be more directed :)
> >
> > Kind regards,
> >
> > Liam Clarke-Hutchinson
> >
> > On Sun, Aug 9, 2020 at 11:43 PM Dor Ben Dov <dor.ben-dov@amdocs.com>
> > wrote:
> >
> > > Hi All,
> > > What is the best recommended way, and tool to backup kafka in
> production?
> > > Regards,
> > > Dor
> > >
> > > This email and the information contained herein is proprietary and
> > > confidential and subject to the Amdocs Email Terms of Service, which
> you
> > > may review at https://www.amdocs.com/about/email-terms-of-service <
> > > https://www.amdocs.com/about/email-terms-of-service>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message