kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gene Robichaux <Gene.Robich...@match.com>
Subject RE: Backups
Date Tue, 20 Jan 2015 14:03:57 GMT
Thanks for the feedback.

Our DEV team has built a MirrorMaker-like process that mirrors all topics between two DCs.
It is basically separate consumer/producer process that shovels data from DC A to DC B between
two separate Kafka clusters.So in essence we have a replication factor of 6 (3 replicas in
each DC). I guess I just need to get over my paranoia :)

Gene Robichaux
Manager, Database Operations
8300 Douglas Avenue I Suite 800 I Dallas, TX  75225

-----Original Message-----
From: Gwen Shapira [mailto:gshapira@cloudera.com] 
Sent: Tuesday, January 20, 2015 12:39 AM
To: users@kafka.apache.org
Subject: Re: Backups


As a former DBA, I hear you on backups :)

Technically, you could copy all log.dir files somewhere safe occasionally. I'm pretty sure
we don't guarantee the consistency or safety of this copy. You could find yourself with a
corrupt "backup"
by copying files that are either in the middle of getting written or are inconsistent in time
with other files. Kafka doesn't have a good way to stop writing to files for long enough to
allow copying them safely.

Unlike traditional backups, there's no transaction log that can be rolled to move a disk copy
forward in time (or that can be used when data files are locked for backups). In Kafka, the
files *are* the transaction log and you roll back in time by deciding which offsets to read.

DR is possible using MirrorMaker though, since the only thing better than replication is...
more replication!
So you could create a non-corrupt file copy by stopping a MirrorMaker replica occasionally
and copying all files somewhere safe.

If it helps you sleep better at night :) Typically having kafka nodes on multiple racks and
a DR in another data center is considered pretty safe.


On Wed, Jan 14, 2015 at 9:22 AM, Gene Robichaux <Gene.Robichaux@match.com> wrote:
> Does anyone have any thoughts on Kafka broker backups?
> All of our topics have a replication factor of 3. However I just want to know if anyone
does anything about traditional backups. My background is Ops DBA, so I have a special place
in my heart for backups.
> Gene Robichaux
> Manager, Database Operations
> Match.com
> 8300 Douglas Avenue I Suite 800 I Dallas, TX  75225
View raw message