kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Loddengaard <a...@confluent.io>
Subject Re: Backing up Kafka data and using it later?
Date Tue, 10 May 2016 22:32:17 GMT
You may find this interesting, although I don't believe it's exactly what
you're looking for:

https://github.com/pinterest/secor

I'm not sure how stable and commonly used it is.

Additionally, I see a lot of users use MirrorMaker for a "backup," where
MirrorMaker copies all topics from one Kafka cluster to another "backup"
cluster. I put "backup" in quotes because this architecture doesn't support
snapshotting like a traditional backup would. I realize this doesn't
address your specific use case, but thought you may find it interesting
regardless.

Sorry I'm a little late to the thread, too.

Alex

On Thu, May 5, 2016 at 7:05 AM, Rad Gruchalski <radek@gruchalski.com> wrote:

> John,
>
> I’m not as expert expert in Kafka but I would assume so.
>
>
>
>
>
>
>
>
>
>
> Best regards,
> Radek Gruchalski
> radek@gruchalski.com (mailto:radek@gruchalski.com) (mailto:
> radek@gruchalski.com)
> de.linkedin.com/in/radgruchalski/ (
> http://de.linkedin.com/in/radgruchalski/)
>
> Confidentiality:
> This communication is intended for the above-named person and may be
> confidential and/or legally privileged.
> If it has come to you in error you must take no action based on it, nor
> must you copy or show it to anyone; please delete/destroy and inform the
> sender immediately.
>
>
>
> On Thursday, 5 May 2016 at 01:46, John Bickerstaff wrote:
>
> > Thanks - does that mean that the only way to safely back up Kafka is to
> > have replication?
> >
> > (I have done this partially - I can get the entire topic on the command
> > line, after completely recreating the server, but my code that is
> intended
> > to do the same thing just hangs)
> >
> > On Wed, May 4, 2016 at 3:18 PM, Rad Gruchalski <radek@gruchalski.com
> (mailto:radek@gruchalski.com)> wrote:
> >
> > > John,
> > >
> > > I believe you mean something along the lines of:
> > > http://markmail.org/message/f7xb5okr3ujkplk4
> > > I don’t think something like this has been done.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > Best regards,
> > > Radek Gruchalski
> > > radek@gruchalski.com (mailto:radek@gruchalski.com) (mailto:
> > > radek@gruchalski.com (mailto:radek@gruchalski.com))
> > > de.linkedin.com/in/radgruchalski/ (
> http://de.linkedin.com/in/radgruchalski/) (
> > > http://de.linkedin.com/in/radgruchalski/)
> > >
> > > Confidentiality:
> > > This communication is intended for the above-named person and may be
> > > confidential and/or legally privileged.
> > > If it has come to you in error you must take no action based on it, nor
> > > must you copy or show it to anyone; please delete/destroy and inform
> the
> > > sender immediately.
> > >
> > >
> > >
> > > On Wednesday, 4 May 2016 at 23:04, John Bickerstaff wrote:
> > >
> > > > Hi,
> > > >
> > > > I have what is probably an edge use case. I'd like to back up a
> single
> > > > Kafka instance such that I can recreate a new server, drop Kafka in,
> drop
> > > > the data in, start Kafka -- and have all my data ready to go again
> for
> > > > consumers.
> > > >
> > > > Is such a thing done? Does anyone have any experience trying this?
> > > >
> > > > I have, and I've run into some problems which suggest there's a
> setting
> > > or
> > > > some other thing I'm unaware of...
> > > >
> > > > If you like, don't think of it as a backup problem so much as a
> "cloning"
> > > > problem. I want to clone a new Kafka machine without actually
> cloning it
> > > >
> > >
> > > -
> > > > I.E. the data is somewhere else (log and index files) although
> Zookeeper
> > >
> > > is
> > > > up and running just fine.
> > > >
> > > > Thanks
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message