kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Su She <suhsheka...@gmail.com>
Subject Re: Consuming Kafka Messages Inside of EC2 Instances
Date Thu, 29 Jan 2015 06:56:21 GMT
Thank you Dillian and Guozhang for the responses.

Yes, Dillian you are understanding my issue correctly. I am not sure what
the best approach to this is...I'm not sure if there's a way to whitelist
certain IPs, create a VPC, use the cluster launcher as the kafka
zookeeper/broker. I guess this is more of an AWS question, but I thought
this is a problem some Kafka users must have solved already.

Edit: I just tried using the cluster launcher as an intermediate. I started
Zookeeper/Kafka Server on my Cluster launcher and then created a
topic/produced messages. I set up a kafka consumer on one of my private EC2
instances, but I got a No Route to host error. I pinged the cluster
launcher <-> private instance and it works fine. I was hoping I could use
this is as a temporary solution...any suggestions on this issue would also
be greatly appreciated. Thanks!

Best,

Su


On Wed, Jan 28, 2015 at 9:11 PM, Guozhang Wang <wangguoz@gmail.com> wrote:

> Su,
>
> Does this help for your case?
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ
>
> Guozhang
>
> On Wed, Jan 28, 2015 at 3:36 PM, Dillian Murphey <crackshotmule@gmail.com>
> wrote:
>
> > Am I understanding your question correctly... You're asking how do you
> > establish connectivity to an instance in a private subnet from the
> outside
> > world?  Are you thinking in terms of zookeeper or just general aws
> network
> > connectivity?
> >
> > On Wed, Jan 28, 2015 at 11:03 AM, Su She <suhshekar52@gmail.com> wrote:
> >
> > > Hello All,
> > >
> > > I have set up a cluster of EC2 instances using this method:
> > >
> > >
> > >
> >
> http://blogs.aws.amazon.com/bigdata/post/Tx2D0J7QOVRJBRX/Deploying-Cloudera-s-Enterprise-Data-Hub-on-AWS
> > >
> > > As you can see the instances are w/in a private subnet. I was wondering
> > if
> > > anyone has any advice on how I can set up a Kafka zookeeper/server on
> an
> > > instance that receives messages from a Kafka Producer outside of the
> > > private subnet. I have tried using the cluster launcher, but I feel
> like
> > it
> > > is not a best practice and only a temporary situation.
> > >
> > > Thank you for the help!
> > >
> > > Best,
> > >
> > > Su
> > >
> >
>
>
>
> --
> -- Guozhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message