kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dillian Murphey <crackshotm...@gmail.com>
Subject Re: Kafka behind AWS ELB
Date Mon, 04 May 2015 20:52:21 GMT
I'm interested in this topic as well.  If you put kafka brokers inside an
autoscaling group, then AWS will automatically add brokers if demand
increases, and the ELB will automatically round-robin across all of your
kafka instances.  So in your config files and code, you only need to
provide a single DNS name (the load balancer). You don't need to specify
all your kafka brokers inside your config file.  If a broker dies, the ELB
will only route to healthy nodes.

So you get a lot of robustness, scalability, and fault-tolerance by using
the AWS services. Kafka Brokers will automatically load balance, but the
question is whether it is ok to put all your brokers behind an ELB and
expect the system to work properly.

What alternatives are there to dynamic/scalable broker clusters?  I don't
want to have to modify my config files or code if I add more brokers, and I
want to be able to handle a broker going down. So these are the reasons AWS
questions like this come up.

Thanks for any comments too. :)




On Mon, May 4, 2015 at 9:03 AM, Mayuresh Gharat <gharatmayuresh15@gmail.com>
wrote:

> Ok. You can deploy kafka in AWS. You can have brokers on AWS servers.
> Kafka is not a push system. So you will need someone writing to kafka and
> consuming from kafka. It will work. My suggestion will be to try it out on
> a smaller instance in AWS and see the effects.
>
> As I do not know the actual use case about why you want to use kafka for, I
> cannot comment on whether it will work for you personalized use case.
>
> Thanks,
>
> Mayuresh
>
> On Mon, May 4, 2015 at 8:55 AM, Chandrashekhar Kotekar <
> shekhar.kotekar@gmail.com> wrote:
>
> > I am sorry but I cannot reveal those details due to confidentiality
> issues.
> > I hope you understand.
> >
> >
> > Regards,
> > Chandrash3khar Kotekar
> > Mobile - +91 8600011455
> >
> > On Mon, May 4, 2015 at 9:18 PM, Mayuresh Gharat <
> > gharatmayuresh15@gmail.com>
> > wrote:
> >
> > > Hi Chandrashekar,
> > >
> > > Can you please elaborate the use case for Kafka here, like how you are
> > > planning to use it.
> > >
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar <
> > > shekhar.kotekar@gmail.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I am new to Apache Kafka. I have played with it on my laptop.
> > > >
> > > > I want to use Kafka in AWS. Currently we have tomcat web servers
> based
> > > REST
> > > > API. We want to replace REST API with Apache Kafka, web servers are
> > > behind
> > > > ELB.
> > > >
> > > > I would like to know if we can keep Kafka brokers behind ELB? Will it
> > > work?
> > > >
> > > > Regards,
> > > > Chandrash3khar Kotekar
> > > > Mobile - +91 8600011455
> > > >
> > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
>
>
>
> --
> -Regards,
> Mayuresh R. Gharat
> (862) 250-7125
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message