kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Different free space and log files between nodes
Date Thu, 02 Oct 2014 19:59:43 GMT
Hello Dayo,

This is a known issue, since today Kafka's log rolling / cleaning policy
depends on the creation timestamp of the segment files, which could be
modified upon partition migration / broker restart, it can cause the server
to not honor the specified log cleaning config. Some more details ca be
found at

KAFKA-881
KAFKA-979

Guozhang


On Thu, Oct 2, 2014 at 10:48 AM, Dayo Oliyide <dayo.oliyide@gmail.com>
wrote:

> Hi,
>
> I've noticed an interesting behaviour which I hope someone can fully
> explain.
> I have 3 Kafka Node cluster with a setting of log.retention.hours=168 (7
> days) and log.segment.bytes=536870912.
>
> I recently restarted one of the nodes and it's uptime is now 3 days behind
> than the other 2.
>
> After about 7 days I noticed that the other 2 nodes cleared out an equal
> amount of stale logs/data,
> but the restarted node didn't cleared out the same amount. The restarted
> node only cleared out a
> similar amount 3 days later. Generally now the restarted node seems to be 3
> days (free space wise) behind
> the other 2.
>
> I noticed that certain partition log and index files on the restarted node
> are different to the other 2.
> Below is an example (Node C is the restarted one)
>
>
> Node A - Topic Z Partition 12
> [        488     Sep 26 11:47]  00000000000000046460.index
> [    1781829  Sep 24 12:09]  00000000000000046460.log
> [   10485760 Sep 28 22:23]  00000000000000046522.index
> [    1536693  Sep 28 22:23]  00000000000000046522.log
>
> Node B - Topic Z Partition 12
> [        488     Sep 26 11:47]  00000000000000046460.index
> [    1781829  Sep 24 12:09]  00000000000000046460.log
> [   10485760 Sep 28 22:23]  00000000000000046522.index
> [    1536693  Sep 28 22:23]  00000000000000046522.log
>
> Node C - Topic Z Partition 12
> [   10485760 Sep 28 22:23]  00000000000000046485.index
> [    2277311  Sep 28 22:23]  00000000000000046485.log
>
> I can see that Node C's base offset (log prefix) is in between the offsets
> of the other logs on Node A & B, and that
> suggests to me some partition 12 messages are on Node A & B but not on Node
> C?
>
> I was hoping someone could help me figure out what's happening.
>
> Thanks
> Dayo
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message