kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Dilemma - ZK consumer problems - upgrade to 0.8?
Date Fri, 07 Jun 2013 19:32:00 GMT
Which version of ZK are you using ?


On Fri, Jun 7, 2013 at 12:15 PM, Evan Chan <ev@ooyala.com> wrote:

> Hey guys,
>
> I sincerely apologize if this has been covered before, I haven't quite
> found a similar situation.
>
> We are using Kafka 0.7.2 in production, and we are using the ZK high level
> Scala consumer.   However, we find the ZK consumer very unstable.  It would
> work for one or two weeks, then suddenly it would complain about ZK nodes
> disappearing, and one consumer would die, then another, then another, until
> our pipeline is no longer pulling any data.   There are multiple
> NullPointerExceptions, and other problems.    We can restart it, but it
> does not stay up predictably.
>
> On the other hand, I have a simple app which I wrote using the simple
> consumer to mirror select partitions (will blog about this later) and it
> just works flawlessly.
>
> So we are faced with a dilemma to get back on track:
> 1)  Use SimpleConsumer, and write our own balancing code  (but honestly our
> boxes almost never go down, compared to the rate of ZK mishaps)
> 2)  Upgrade to Kafka 0.8 and hope that that resolves the issue.
>
> There seem to be so many improvements in 0.8 that that seems to be the
> biggest win long-term, so I am wondering if people can comment on:
> - has anyone tried using 0.8 in production?  Is it stable yet?
> - How much more stable is the ZK consumer in 0.8?
> - will it be possible to change the offset in the 0.8 consumer?  That was
> the other reason why we wanted to move to SimpleConsumer.
>
> thanks,
> Evan
>
> --
> --
> Evan Chan
> Staff Engineer
> ev@ooyala.com  |
>
> <http://www.ooyala.com/>
> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><
> http://www.twitter.com/ooyala>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message