kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Brown <...@metamx.com>
Subject Re: Kafka consumer not reconnecting to restarted Kafka node.
Date Tue, 25 Oct 2011 18:52:45 GMT
So would you all accept a patch that exposes the zk paths in the config?

 Dan

On Wed, Oct 19, 2011 at 14:20, Dan Brown <dan@metamx.com> wrote:
> No, configurable kafka paths would be subsumed by a properly
> functioning zk chroot. I can't imagine a situation where the brokers
> and consumers could be safely decoupled.
>
>  Dan
>
> On Wed, Oct 19, 2011 at 13:19, Jay Kreps <jay.kreps@gmail.com> wrote:
>> Adding a manual override path in the config seems like an acceptable thing
>> to do, but it is not clear we will get 0.8 out before zookeeper gets their
>> next release out. Would this have any use other than working around this zk
>> problem?
>>
>> -Jay
>>
>> On Wed, Oct 19, 2011 at 1:02 PM, Mathias Herberts <
>> mathias.herberts@gmail.com> wrote:
>>
>>> Not using a chroot is not a feasible thing when the ZK ensemble is
>>> shared by several Kafka clusters.
>>>
>>> On Wed, Oct 19, 2011 at 21:54, Jun Rao <junrao@gmail.com> wrote:
>>> > This particular ZK issue only exposes if the ZK connect string uses
>>> chroot
>>> > and there is a ZK disconnect. In the near term, you can choose not to use
>>> > chroot in your ZK connect string.
>>> >
>>> > Thanks,
>>> >
>>> > Jun
>>> >
>>> > On Wed, Oct 19, 2011 at 12:30 PM, Dan Brown <dan@metamx.com> wrote:
>>> >
>>> >> On Tue, Oct 18, 2011 at 15:02, Mathias Herberts
>>> >> <mathias.herberts@gmail.com> wrote:
>>> >> >> I think Jun or Neha had tracked this one down at LinkedIn and
it is
>>> >> actually
>>> >> >> a zk bug. I think it is this one, but they could confirm:
>>> >> >>
>>> >> >> https://issues.apache.org/jira/browse/ZOOKEEPER-961
>>> >> >
>>> >> > Sure looks like it.
>>> >>
>>> >> Confirmed. Running the above repro against zookeeper trunk produces
a
>>> >> responsive consumer that successfully rebalances when the broker
>>> >> rejoins.
>>> >>
>>> >> > So besides waiting for ZK 3.3.4 there is not much that can be done
>>> >> > since Kafka does not offer a way to change the root znode of a
>>> >> > cluster.
>>> >>
>>> >> Yeah, it's unclear that zk-3.3.4 will release anytime soon. Instead
of
>>> >> relying on zk chroot, why not just expose the zk paths as config
>>> >> parameters for consumers, brokers, and producers?
>>> >>
>>> >> object ZkUtils {
>>> >>  val ConsumersPath = "/consumers"
>>> >>  val BrokerIdsPath = "/brokers/ids"
>>> >>  val BrokerTopicsPath = "/brokers/topics"
>>> >>  ...
>>> >>
>>> >
>>>
>>
>

Mime
View raw message