kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Dilemma - ZK consumer woes - upgrade to 0.8?
Date Sat, 08 Jun 2013 04:05:05 GMT
Which version of ZK are you using?

Thanks,

Jun


On Fri, Jun 7, 2013 at 12:20 PM, Evan Chan <ev@ooyala.com> wrote:

> [ Sorry if this mail is duplicated, this is my fourth try sending this
> message]
>
> Hey guys,
>
> I sincerely apologize if this has been covered before, I haven't quite
> found a similar situation.
>
> We are using Kafka 0.7.2 in production, and we are using the ZK high level
> Scala consumer.   However, we find the ZK consumer very unstable.  It would
> work for one or two weeks, then suddenly it would complain about ZK nodes
> disappearing, and one consumer would die, then another, then another, until
> our pipeline is no longer pulling any data.   There are multiple
> NullPointerExceptions, and other problems.    We can restart it, but it
> does not stay up predictably.
>
> On the other hand, I have a simple app which I wrote using the simple
> consumer to mirror select partitions (will blog about this later) and it
> just works flawlessly.
>
> So we are faced with a dilemma to get back on track:
> 1)  Use SimpleConsumer, and write our own balancing code  (but honestly our
> boxes almost never go down, compared to the rate of ZK mishaps)
> 2)  Upgrade to Kafka 0.8 and hope that that resolves the issue.
>
> There seem to be so many improvements in 0.8 that that seems to be the
> biggest win long-term, so I am wondering if people can comment on:
> - has anyone tried using 0.8 in production?  Is it stable yet?
> - How much more stable is the ZK consumer in 0.8?
> - will it be possible to change the offset in the 0.8 consumer?  That was
> the other reason why we wanted to move to SimpleConsumer.
>
> thanks,
> Evan
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message