kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charity Majors <char...@hound.sh>
Subject sarama and __consumer_offsets rebalancing
Date Sun, 03 Jul 2016 01:47:57 GMT
Hi there,

I'm trying to track down why __consumer_offsets is never balancing and what
to do about it.

We're using Shopify's sarama
<https://gowalker.org/github.com/Shopify/sarama> client, which explicitly
states that it doesn't do rebalancing for internal topics ... ever.

It seems like sarama is one of the better-regarded golang clients for
kafka, despite this fairly glaring and confusing oversight.  (is this
true?  does anyone have a better recommendation?)

There is a sarama-cluster <https://github.com/bsm/sarama-cluster> extension
with a CLI.  It looks like if you compile the CLI and connect to the
cluster, it seems it will kick off a rebalance and remove downed brokers.
This ... seems incredibly awkward to me, but at least it's *a* solution.
But it only works if you're using consumer groups.  Which we are not.

We can rewrite the kafka consumer to use consumer groups, though after
reading docs for a day I still don't understand why we would do that other
than to make sarama-cluster work, or what the impact would be other than
letting us use this thing that lets us rebalance.

Last question.  We are using kafka-0.9, and we seem to keep running into
shit that is not very well documented and doesn't seem to be used by very
many people.  I'm thinking of downgrading to 0.8 because none of the 0.9
features are very compelling to us, and we would really not rather be
blazing any unnecessary trails.  Does anyone have an informed opinion here?

Thanks for answering all the noob questions.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message