kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charlie Hull <char...@flax.co.uk>
Subject Re: Default behavior for full broker
Date Thu, 21 Apr 2016 14:19:08 GMT
On 21/04/2016 15:16, Tom Crayford wrote:
> Generally, Kafka will crash when the disk fills up (it gets an exception
> trying to do the write to the disk). That will indeed lead to the behaviour
> you've talked about where all the brokers end up crashing, and there's no
> contingency for it in the codebase right now (and generally writing code
> for such scenarios is very tricky). The real trick is to monitor and ensure
> you don't run out of disk space at all.

We've been working on a project using Kafka and the ELK stack and run 
into exactly this problem - although not with Kafka but with 
Elasticsearch. Hope the following is of interest! I'm not sure how you 
might apply the same to Kafka myself but perhaps there's a way...
http://www.flax.co.uk/blog/2016/04/21/running-disk-space-elasticsearch-solr/

Cheers

Charlie
>
> On Thu, Apr 21, 2016 at 3:11 PM, Jens Rantil <jens.rantil@tink.se> wrote:
>
>> Hi Lawrence,
>>
>>> As I'm thinking through this a little more, if that is the case and the
>> node is removed, some partitions in the system may be marked as
>> under-replicated and cause a cascading effect where partitions are
>> re-replicated and cause other nodes to fill up.  Has that ever happened?
>> Does Kafka have a contingency plan for such a scenario?
>>
>> Currently, Kafka doesn't rebalance partitions automagically if there is an
>> issue with a broker. That excludes the failure scenario that you portrait.
>>
>> Regarding how Kafka handles a full disk, I can't answer that.
>>
>> Cheers,
>> Jens
>>
>> On Wed, Apr 20, 2016 at 7:09 PM Lawrence Weikum <lweikum@pandora.com>
>> wrote:
>>
>>> Hello,
>>>
>>> I'm curious about the expected or default behavior that might occur if a
>>> broker in the system has filled up.  By that I mean when a broker has
>> used
>>> all of its memory and disk space.  Is the node simply removed from the
>>> system until space is cleared?
>>>
>>> As I'm thinking through this a little more, if that is the case and the
>>> node is removed, some partitions in the system may be marked as
>>> under-replicated and cause a cascading effect where partitions are
>>> re-replicated and cause other nodes to fill up.  Has that ever happened?
>>> Does Kafka have a contingency plan for such a scenario?
>>>
>>> Thank you so much for your insight and all of your hard work!
>>>
>>> Lawrence
>>>
>> --
>>
>> Jens Rantil
>> Backend Developer @ Tink
>>
>> Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
>> For urgent matters you can reach me at +46-708-84 18 32.
>>
>


-- 
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk

Mime
View raw message