kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <joe.st...@stealth.ly>
Subject Re: kafka logs gone after reboot the server
Date Sat, 03 Jan 2015 00:47:53 GMT
Either should be fine but is all dependent on your environment and how you
want to operate your cluster in the long run i.e. if you have multiple
volumes then you need to have them each in the log.dirs separated by comma,
etc. If there is no good reason to have the broker.id in the directory they
you should probably have it be consistent on each node.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/

On Fri, Jan 2, 2015 at 5:26 PM, Sa Li <salicn@gmail.com> wrote:

> One more question, when I set the log.dirs in different nodes in the
> cluster, should I set them different name, say kafka-logs-1 which
> associated with broker id, or I can set the same directory name, like
> /var/log/kafka for every node (assume one broker in each server).
>
> thanks
>
>
> On Fri, Jan 2, 2015 at 2:20 PM, Sa Li <salicn@gmail.com> wrote:
>
> > Thanks a lot!
> >
> >
> > On Fri, Jan 2, 2015 at 12:15 PM, Jay Kreps <jay.kreps@gmail.com> wrote:
> >
> >> Nice catch Joe--several people have complained about this as a problem
> and
> >> we were a bit mystified as to what kind of bug could lead to all their
> >> logs
> >> getting deleted and re-replicated when they bounced the server. We
> assumed
> >> "bounced" meant restarted the app, but I think likely what is happening
> is
> >> what you describe--the logs were in /tmp and bouncing the server meant
> >> restarting.
> >>
> >> -Jay
> >>
> >> On Fri, Jan 2, 2015 at 11:02 AM, Joe Stein <joe.stein@stealth.ly>
> wrote:
> >>
> >> > That is because your logs are in /tmp which you can change by
> >> > setting log.dirs to something else.
> >> >
> >> > /*******************************************
> >> >  Joe Stein
> >> >  Founder, Principal Consultant
> >> >  Big Data Open Source Security LLC
> >> >  http://www.stealth.ly
> >> >  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> >> > ********************************************/
> >> >
> >> > On Fri, Jan 2, 2015 at 1:58 PM, Sa Li <salicn@gmail.com> wrote:
> >> >
> >> > > Hi, All
> >> > >
> >> > > I've just notice one thing, when I am experiencing some errors in
> >> Kafka
> >> > > servers, I reboot the dev servers (not a good way), after reboot,
I
> >> get
> >> > > into zkCli, I can see all the topics still exist. But when I get
> into
> >> > kafka
> >> > > log directory, I found all data gone, see
> >> > >
> >> > > root@DO-mq-dev:/tmp/kafka-logs-1/ui_test_topic_4-0# ll
> >> > > total 8
> >> > > drwxr-xr-x  2 root root     4096 Jan  2 09:39 ./
> >> > > drwxr-xr-x 46 root root     4096 Jan  2 10:46 ../
> >> > > -rw-r--r--  1 root root 10485760 Jan  2 09:39
> >> 00000000000000000000.index
> >> > > -rw-r--r--  1 root root        0 Jan  2 09:39
> 00000000000000000000.log
> >> > >
> >> > > I wonder, if for some reasons, the server down, and restart it, all
> >> the
> >> > > data in hard drive will be gone?
> >> > >
> >> > > thanks
> >> > >
> >> > > --
> >> > >
> >> > > Alec Li
> >> > >
> >> >
> >>
> >
> >
> >
> > --
> >
> > Alec Li
> >
>
>
>
> --
>
> Alec Li
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message