samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: Collocating Samza(YARN) and Kafka/ZK clusters
Date Tue, 10 Feb 2015 00:54:16 GMT
It may or may not be due to colocating Kafka and Samza but you are probably
tripping the failure detection in Kafka which considers a replica out of
sync if it falls more than N messages behind. Can you try tuning this
setting as described here:
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowtoreducechurnsinISR?WhendoesabrokerleavetheISR
?

-Jay

On Mon, Feb 9, 2015 at 4:35 PM, Karthik Sriram <amaron.me@gmail.com> wrote:

> Hey all,
>  I'm trying to run samza on a 5 node (YARN/Kafka/ZK) cluster with each box
> running all 3 processes on AWS. I have been facing very weird performance
> issues with Kafka when run this way. Kafka seems to get unbalanced very
> often with replicas going out of sync every so often. This results in lost
> messages when producing to this cluster. I initially suspected it was a
> scale issue (70k-80k qps of incoming messages, ~120k qps peak) and reduced
> write throughput by sampling just 10% of the messages but I still noticed
> the same issues. The weird part is that this doesn't happen every time I
> run, but many of the times.
>
> We have been using a much larger Kafka cluster for long with great
> performance and have never seen such issues before. Then I saw (
> https://engineering.linkedin.com/samza/operating-apache-samza-scale) which
> mentions that LinkedIn also faced some issues when collocating Samza and
> Kafka.
>
> Can someone throw some light on this? Is collocating samza and kafka a
> strict no, or is it more likely a Kafka/machine tuning issue ? Any help is
> appreciated!
>
> Kafka version : 0.8.1.1
> Samza version: 0.8
>
> Thanks a lot for your time,
> Karthik
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message