kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Davis <davi...@gmail.com>
Subject Re: Lots of messages when starting Kafka
Date Wed, 06 Jul 2016 14:16:34 GMT
 I've seen Kafka take much longer than 10 seconds to stop normally. (I
don't use docker.)

controlled.shutdown.enable (true)
Enable controlled shutdown of the server

controlled.shutdown.max.retries (3)
Controlled shutdown can fail for multiple reasons. This determines the
number of retries when such failure happens

controlled.shutdown.retry.backoff.ms (5000)
Before each retry, the system needs time to recover from the state that
caused the previous failure (Controller fail over, replica lag etc). This
config determines the amount of time to wait before retrying.

-Peter

On Wed, Jul 6, 2016 at 07:09 Fabian Kochem <
fabian.kochem@concepts-and-training.de> wrote:

> I use "docker stop" which sends SIGTERM and then waits 10 seconds before
> it sends SIGKILL.
> I assume it doesn't get to send SIGKILL because when I don't use Docker,
> a usual Ctrl+C stops Kafka way faster than 10 seconds.
>
>
> On 06/07/16 16:05, Peter Davis wrote:
> > You're right Sam. Getting my signals confused before coffee.
> >
> > Fabian, how did you in fact kill Kafka? "kill"? Ctrl+C?
> >
> > -Peter
> >
> > On Wed, Jul 6, 2016 at 07:00 Sam Pegler <sam.pegler@infectiousmedia.com>
> > wrote:
> >
> >>> If you kill Kafka with SIGTERM, then naturally it will leave behind
> >> corrupt
> >> index files. It will safely rebuild them. Don't worry.
> >>
> >> I think you mean SIGKILL.  SIGTERM should be handled normally an won't
> >> result in corrupted index files at startup.
> >>
> >> __
> >>
> >> Sam Pegler
> >>
> >> Production Engineer
> >>
> >> T. +44(0) 07 562 867 486
> >>
> >> <http://www.infectiousmedia.com/>
> >> 3-7 Herbal Hill / London / EC1R 5EJ
> >> www.infectiousmedia.com
> >>
> >> This email and any attachments are confidential and may also be
> privileged.
> >> If you
> >> are not the intended recipient, please notify the sender immediately,
> and
> >> do not
> >> disclose the contents to another person, use it for any purpose, or
> store,
> >> or copy
> >> the information in any medium. Please also destroy and delete the
> message
> >> from
> >> your computer.
> >>
> >>
> >> On 6 July 2016 at 14:54, Peter Davis <davispw@gmail.com> wrote:
> >>
> >>> If you kill Kafka with SIGTERM, then naturally it will leave behind
> >> corrupt
> >>> index files. It will safely rebuild them. Don't worry.
> >>>
> >>> The Group Metadata Manager messages are normal -- that's just Kafka
> >>> starting up.
> >>>
> >>> Perhaps the corrupted index log messages could be less scary, though.
> >> This
> >>> is a very common question from new users...
> >>>
> >>> -Peter
> >>>
> >>> On Wed, Jul 6, 2016 at 06:50 Fabian Kochem <
> >>> fabian.kochem@concepts-and-training.de> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> this is my first time using a mailing list so please forgive me if I'm
> >>>> doing something wrong.
> >>>>
> >>>> When I start Kafka, I get a ton of messages like this:
> >>>>
> >>>> WARN Found a corrupted index file,
> >>>> /kafka/logs/__consumer_offsets-48/00000000000000000000.index, deleting
> >>>> and rebuilding index... (kafka.log.Log)
> >>>> INFO Recovering unflushed segment 0 in log __consumer_offsets-48.
> >>>> (kafka.log.Log)
> >>>> INFO Completed load of log __consumer_offsets-48 with log end offset
0
> >>>> (kafka.log.Log)
> >>>> WARN Found a corrupted index file,
> >>>> /kafka/logs/__consumer_offsets-19/00000000000000000000.index, deleting
> >>>> and rebuilding index... (kafka.log.Log)
> >>>> INFO Recovering unflushed segment 0 in log __consumer_offsets-19.
> >>>> (kafka.log.Log)
> >>>> INFO Completed load of log __consumer_offsets-19 with log end offset
0
> >>>> (kafka.log.Log)
> >>>> WARN Found a corrupted index file,
> >>>> /kafka/logs/__consumer_offsets-3/00000000000000000000.index, deleting
> >>>> and rebuilding index... (kafka.log.Log)
> >>>> INFO Recovering unflushed segment 0 in log __consumer_offsets-3.
> >>>> (kafka.log.Log)
> >>>> INFO Completed load of log __consumer_offsets-3 with log end offset
0
> >>>> (kafka.log.Log)
> >>>>
> >>>> And after that, there's an equal amount of messages which look like
> >> this:
> >>>>
> >>>> INFO [Group Metadata Manager on Broker 1001]: Loading offsets and
> group
> >>>> metadata from [__consumer_offsets,22]
> >>>> (kafka.coordinator.GroupMetadataManager)
> >>>> INFO [Group Metadata Manager on Broker 1001]: Finished loading offsets
> >>>> from [__consumer_offsets,22] in 11 milliseconds.
> >>>> (kafka.coordinator.GroupMetadataManager)
> >>>> INFO [Group Metadata Manager on Broker 1001]: Loading offsets and
> group
> >>>> metadata from [__consumer_offsets,25]
> >>>> (kafka.coordinator.GroupMetadataManager)
> >>>> INFO [Group Metadata Manager on Broker 1001]: Finished loading offsets
> >>>> from [__consumer_offsets,25] in 4 milliseconds.
> >>>> (kafka.coordinator.GroupMetadataManager)
> >>>> INFO [Group Metadata Manager on Broker 1001]: Loading offsets and
> group
> >>>> metadata from [__consumer_offsets,28]
> >>>> (kafka.coordinator.GroupMetadataManager)
> >>>> INFO [Group Metadata Manager on Broker 1001]: Finished loading offsets
> >>>> from [__consumer_offsets,28] in 2 milliseconds.
> >>>> (kafka.coordinator.GroupMetadataManager)
> >>>>
> >>>> Did I misconfigure something?
> >>>>
> >>>> I only have one topic with a few consumers, so this seems a bit much.
> I
> >>>> shutdown Kafka using SIGTERM.
> >>>>
> >>>> Thanks in advance,
> >>>> Fabian Kochem
> >>>>
> >>>
> >>
> >
>
> --
> Fabian Kochem
> Entwicklung
>
> ---------------------------------------------
> CaT Concepts and Training GmbH
> Vorgebirgstraße 338
> 50969 Köln
>
> Fon : +49 (0) 221 / 46 75 76 - 53
> Fax : +49 (0) 221 / 46 75 76 - 09
> ---------------------------------------------
> Geschäftsführung:
>
> Sven Kapust, Gerald Konrad, Alexandra Oehlke,
> Sandra Röbbelen, Volker Röbbelen, Denis Witt
>
> Amtsgericht Köln HRB 57804
> Steuer-Nr.: 217/5716/0769
> Ust-ID-Nr.: DE 814694228
> Sitz: Köln
> ---------------------------------------------
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message