kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Brown <...@metamx.com>
Subject Re: Kafka consumer not reconnecting to restarted Kafka node.
Date Tue, 25 Oct 2011 23:07:57 GMT
Hi Jun, zk-3.4.0 (or zk-3.3.4) will fix my problem, which is just the chroot
issue.

 Dan

On Tue, Oct 25, 2011 at 12:22, Jun Rao <junrao@gmail.com> wrote:

> Dan,
>
> My understanding is that ZK 3.4 will be released soon, which will fix the
> chroot issue. Will that fix your problem?
>
> Thanks,
>
> Jun
>
> On Tue, Oct 25, 2011 at 11:52 AM, Dan Brown <dan@metamx.com> wrote:
>
> > So would you all accept a patch that exposes the zk paths in the config?
> >
> >  Dan
> >
> > On Wed, Oct 19, 2011 at 14:20, Dan Brown <dan@metamx.com> wrote:
> > > No, configurable kafka paths would be subsumed by a properly
> > > functioning zk chroot. I can't imagine a situation where the brokers
> > > and consumers could be safely decoupled.
> > >
> > >  Dan
> > >
> > > On Wed, Oct 19, 2011 at 13:19, Jay Kreps <jay.kreps@gmail.com> wrote:
> > >> Adding a manual override path in the config seems like an acceptable
> > thing
> > >> to do, but it is not clear we will get 0.8 out before zookeeper gets
> > their
> > >> next release out. Would this have any use other than working around
> this
> > zk
> > >> problem?
> > >>
> > >> -Jay
> > >>
> > >> On Wed, Oct 19, 2011 at 1:02 PM, Mathias Herberts <
> > >> mathias.herberts@gmail.com> wrote:
> > >>
> > >>> Not using a chroot is not a feasible thing when the ZK ensemble is
> > >>> shared by several Kafka clusters.
> > >>>
> > >>> On Wed, Oct 19, 2011 at 21:54, Jun Rao <junrao@gmail.com> wrote:
> > >>> > This particular ZK issue only exposes if the ZK connect string
uses
> > >>> chroot
> > >>> > and there is a ZK disconnect. In the near term, you can choose
not
> to
> > use
> > >>> > chroot in your ZK connect string.
> > >>> >
> > >>> > Thanks,
> > >>> >
> > >>> > Jun
> > >>> >
> > >>> > On Wed, Oct 19, 2011 at 12:30 PM, Dan Brown <dan@metamx.com>
> wrote:
> > >>> >
> > >>> >> On Tue, Oct 18, 2011 at 15:02, Mathias Herberts
> > >>> >> <mathias.herberts@gmail.com> wrote:
> > >>> >> >> I think Jun or Neha had tracked this one down at
LinkedIn and
> it
> > is
> > >>> >> actually
> > >>> >> >> a zk bug. I think it is this one, but they could
confirm:
> > >>> >> >>
> > >>> >> >> https://issues.apache.org/jira/browse/ZOOKEEPER-961
> > >>> >> >
> > >>> >> > Sure looks like it.
> > >>> >>
> > >>> >> Confirmed. Running the above repro against zookeeper trunk
> produces
> > a
> > >>> >> responsive consumer that successfully rebalances when the
broker
> > >>> >> rejoins.
> > >>> >>
> > >>> >> > So besides waiting for ZK 3.3.4 there is not much that
can be
> done
> > >>> >> > since Kafka does not offer a way to change the root znode
of a
> > >>> >> > cluster.
> > >>> >>
> > >>> >> Yeah, it's unclear that zk-3.3.4 will release anytime soon.
> Instead
> > of
> > >>> >> relying on zk chroot, why not just expose the zk paths as
config
> > >>> >> parameters for consumers, brokers, and producers?
> > >>> >>
> > >>> >> object ZkUtils {
> > >>> >>  val ConsumersPath = "/consumers"
> > >>> >>  val BrokerIdsPath = "/brokers/ids"
> > >>> >>  val BrokerTopicsPath = "/brokers/topics"
> > >>> >>  ...
> > >>> >>
> > >>> >
> > >>>
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message