kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liam Clarke-Hutchinson <liam.cla...@adscale.co.nz>
Subject Stuck replicas - can't remove them
Date Mon, 15 Jun 2020 01:35:37 GMT
We hit some memory issues with our brokers caused by large numbers of high
cardinality unique client ids connecting and disconnecting and leaving
behind MBeans (v0.11.0.2 and yep, upgrading is definitely high on the
agenda now) and after a rolling restart, we've got a weird situation.

For topic X, partition 10 shows as having 3 replicas on brokers [A, B, C],
with [B, C] as the ISR - but the thing is, this topic now has a replication
factor of 2, and it's only this one partition that has 3.

I've tried running reassign-partitions for that partition specifying that
it's replicas are only B and C, but the reassignment request fails.

It's causing issues in that our alerting on under-replicated partitions
keeps picking up partition 10 as being underreplicated.

Any ideas on how to convince the cluster to discard the replica on broker A?

Kind regards,

Liam Clarke-Hutchinson

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message