kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: Broker crashes when no space left for log.dirs
Date Thu, 15 Aug 2013 00:06:11 GMT
Interesting - I agree that it is unfortunate we crash in this
scenario. i.e., if the brokers don't crash at least the published
messages remain available for consumption. However, I think this
problem would effectively be subsumed by doing capacity planning ahead
of time and then setting up alerts when thresholds are close to your
limits although that assumes such alerts are reacted to quickly


On Wed, Aug 14, 2013 at 9:52 AM, Bryan Baugher <bjbq4d@gmail.com> wrote:
> Hi,
> This is more of a thought question than a problem that I need support for.
> I have trying out Kafka 0.8.0-beta1 with replication. For our user case we
> want to try and guarantee that our consumers will see all messages even if
> they have fallen greatly behind the broker/producer. For this reason I
> wanted to know how the broker would react when the filesystem it writes its
> messages to is full. What I found was that the broker crashes and cannot be
> started until the filesystem has space again.
> Is there or would it make sense to provide configuration allowing the
> broker to reject writes in this case rather than crashing, electing a new
> leader and attempting the write again? I can clearly understand the use
> case that we don't want to 'lose' messages from the producer and I could
> also see how lack of filesystem space could be considered a machine
> failure, but with replication I would think if you are running out of space
> on 1 broker you are likely running out of space on others.
> Bryan

View raw message