samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gian Merlino <>
Subject Re: Your experience running Samza in AWS?
Date Sat, 14 Feb 2015 16:45:17 GMT
Hi Geoffry,

We've been using Samza in production on AWS for a little over a month.
We're just using the YARN runner on a mostly stock hadoop 2.4.0 cluster
(not EMR). Our experience is that c3s work well for the YARN instances and
i2s work well for the Kafka instances. Things have been pretty solid with
that setup.

For scaling up and scaling down YARN, we just terminate instances or add
instances, and this works pretty well. It can take a few minutes for the
cluster to realize a node has gone and respawn containers elsewhere.

We have a separate Kafka cluster just for Samza's use, different from our
main Kafka cluster. The main reason is that we wanted to isolate off the
disk and network load of state compactions and restores (we don't use
compacted topics in our main Kafka cluster, but we do use them with Samza,
and the extra load on Kafka can be substantial).


On Mon, Feb 9, 2015 at 1:24 PM, Geoffry Sumter <> wrote:

> Hello,
> I'm looking to experiment with Samza more but wanted to get feedback on
> using it in AWS, particularly in production. Are you using YARN? Mesos?
> Something custom? Have you documented tradeoffs you've made, reliability
> concerns, or pitfalls you've discovered? Is there anything you wish you had
> known first? I'd love to benefit from past experience if you have time! :)
> I see "This means that YARN can be replaced with other virtualization
> frameworks — in particular, we are interested in adding direct AWS
> integration. Many companies run in AWS which is itself a virtualization
> framework" from
> Is there work currently being done on this effort?
> Thanks for the help,
> Geoffry

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message