kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Su She <suhsheka...@gmail.com>
Subject Re: Consuming Kafka Messages Inside of EC2 Instances
Date Thu, 29 Jan 2015 17:44:40 GMT
Thank you Guozhang for an updated link.

I read the answer for "Why can't my consumers/producers connect to the
brokers?". I am confused on a couple things. I can ping the private IP of
 A (Cluster Launcher) from B (EC2 instance). I can't ping the public ip of
A (or google.com) from B which makes sense as B is in a private subnet. But
when I try to connect to the zookeeper at private.ip.A:2181 I get a No
Route to Host error. Is there a reason why I can ping private.ip.A but not
connect to private.ip.A:2181?

I have not changed any of my server or producer properties. I have tried to
change advertised host name, but no luck.

Thanks for the help!



On Thu, Jan 29, 2015 at 8:12 AM, Guozhang Wang <wangguoz@gmail.com> wrote:

> Sorry my previous link was not complete:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
> ?
>
>
>
> On Wed, Jan 28, 2015 at 10:56 PM, Su She <suhshekar52@gmail.com> wrote:
>
> > Thank you Dillian and Guozhang for the responses.
> >
> > Yes, Dillian you are understanding my issue correctly. I am not sure what
> > the best approach to this is...I'm not sure if there's a way to whitelist
> > certain IPs, create a VPC, use the cluster launcher as the kafka
> > zookeeper/broker. I guess this is more of an AWS question, but I thought
> > this is a problem some Kafka users must have solved already.
> >
> > Edit: I just tried using the cluster launcher as an intermediate. I
> started
> > Zookeeper/Kafka Server on my Cluster launcher and then created a
> > topic/produced messages. I set up a kafka consumer on one of my private
> EC2
> > instances, but I got a No Route to host error. I pinged the cluster
> > launcher <-> private instance and it works fine. I was hoping I could use
> > this is as a temporary solution...any suggestions on this issue would
> also
> > be greatly appreciated. Thanks!
> >
> > Best,
> >
> > Su
> >
> >
> > On Wed, Jan 28, 2015 at 9:11 PM, Guozhang Wang <wangguoz@gmail.com>
> wrote:
> >
> > > Su,
> > >
> > > Does this help for your case?
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ
> > >
> > > Guozhang
> > >
> > > On Wed, Jan 28, 2015 at 3:36 PM, Dillian Murphey <
> > crackshotmule@gmail.com>
> > > wrote:
> > >
> > > > Am I understanding your question correctly... You're asking how do
> you
> > > > establish connectivity to an instance in a private subnet from the
> > > outside
> > > > world?  Are you thinking in terms of zookeeper or just general aws
> > > network
> > > > connectivity?
> > > >
> > > > On Wed, Jan 28, 2015 at 11:03 AM, Su She <suhshekar52@gmail.com>
> > wrote:
> > > >
> > > > > Hello All,
> > > > >
> > > > > I have set up a cluster of EC2 instances using this method:
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> http://blogs.aws.amazon.com/bigdata/post/Tx2D0J7QOVRJBRX/Deploying-Cloudera-s-Enterprise-Data-Hub-on-AWS
> > > > >
> > > > > As you can see the instances are w/in a private subnet. I was
> > wondering
> > > > if
> > > > > anyone has any advice on how I can set up a Kafka zookeeper/server
> on
> > > an
> > > > > instance that receives messages from a Kafka Producer outside of
> the
> > > > > private subnet. I have tried using the cluster launcher, but I feel
> > > like
> > > > it
> > > > > is not a best practice and only a temporary situation.
> > > > >
> > > > > Thank you for the help!
> > > > >
> > > > > Best,
> > > > >
> > > > > Su
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message