kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Jorgensen <ajorgen...@twitter.com.INVALID>
Subject New broker ignoring retention
Date Mon, 06 Apr 2015 20:10:37 GMT
I can't find any, but does anyone know of any bugs in 0.8.1.1 that would
cause new brokers added to an existing cluster to ignore the per-topic
configuration for retention?

I had a 8 node cluster with a topic with per topic retention set
like: Configs:retention.ms=5400000. I attempted to add 2 more brokers to
the cluster today and transfer 3 of the existing partitions over to the new
nodes. In general, the existing nodes stay around 60% disk usage but the
new brokers start at around 60% and then fall over the course of about two
hours down to 0%. It is unclear to me why the new brokers are ignoring the
log retention time and seemingly keeping the logs indefinitely. Both the
existing brokers and the new ones have the same server.properties file
which the default log.retention.hours=24.

To add the new brokers I ran the reassign-partition tool and just moved 3
partitions from some other nodes to the new nodes, the reassignment seemed
to complete successfully, there are 30 partitions and 10 brokers so each
broker has 3 partitions.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message