kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Su She <suhsheka...@gmail.com>
Subject Re: Consuming Kafka Messages Inside of EC2 Instances
Date Thu, 29 Jan 2015 20:47:27 GMT
Thanks Jonathon!

1) How can I get the whole command string? What i c/p was the whole output

2) I do have Zookeeper as a service from Cloudera Manager, except I used to
have this before as well and was never a problem. Is there a way I can keep
the Zookeeper from CM as well as manually start it up? I don't want to
potentially cause other services depending on the CM zookeeper to fail. Or
maybe I can connect kafka producer to the zookeeper on CM?

Thanks a lot, really appreciate the help.

On Thu, Jan 29, 2015 at 12:42 PM, Jonathan Natkins <natty@streamsets.com>
wrote:

> Hey Su,
>
> It's hard to say, because you didn't copy the entire command string, but it
> looks like it's probably a process that's being managed by Cloudera
> Manager. Do you have a Cloudera Manager instance that is running a
> Zookeeper quorum? If so, that's where the conflict is. You can either
> reconfigure your Zookeeper in CM, or stop the service, and start up a
> Zookeeper manually to try to bind to the public IPs.
>
> Thanks,
> Natty
>
> Jonathan "Natty" Natkins
> StreamSets | Customer Engagement Engineer
> mobile: 609.577.1600 | linkedin <http://www.linkedin.com/in/nattyice>
>
>
> On Thu, Jan 29, 2015 at 12:36 PM, Su She <suhshekar52@gmail.com> wrote:
>
> > Thanks Jonathan!
> >
> > This was the result, would it be okay for me to kill 3544? Or do I try to
> > access a new port?:
> >
> > *sudo netstat -tulpn         | grep 2181*
> >
> > tcp        0      0 0.0.0.0:2181                0.0.0.0:*
> >         LISTEN      3544/java
> >
> >
> > *ps 3544*
> >
> > PID TTY      STAT   TIME COMMAND
> > 3544 ?        Sl     0:19 /usr/java/jdk1.7.0_67-cloudera/bin/java -
> >
> >
> >
> > On Thu, Jan 29, 2015 at 12:31 PM, Jonathan Natkins <natty@streamsets.com
> >
> > wrote:
> >
> > > Hey Su,
> > >
> > > That exception indicates that there's something else already running on
> > the
> > > port that you're trying to start up Zookeeper on. The quickest way to
> > > figure out what's causing the conflict is to run netstat and look for
> the
> > > port:
> > >
> > > [root@ip-10-0-0-45 ~]# *netstat -tulpn | grep 2181*
> > > tcp        0      0 0.0.0.0:2181                0.0.0.0:*
> > > LISTEN      9814/java
> > > [root@ip-10-0-0-45 ~]# *ps 9814*
> > >   PID TTY      STAT   TIME COMMAND
> > >  9814 ?        Sl     8:11 /usr/java/jdk1.7.0_67-cloudera/bin/java -cp
> > >
> > >
> >
> /var/run/cloudera-scm-agent/process/98-zookeeper-server:/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/z
> > >
> > > Jonathan "Natty" Natkins
> > > StreamSets | Customer Engagement Engineer
> > > mobile: 609.577.1600 | linkedin <http://www.linkedin.com/in/nattyice>
> > >
> > >
> > > On Thu, Jan 29, 2015 at 11:47 AM, Su She <suhshekar52@gmail.com>
> wrote:
> > >
> > > > I tried a new approach and have started ec2 instances in a public
> > subnet
> > > > rather than a private subnet. However, now when I try to start
> > zookeeper
> > > I
> > > > get this error. How can I go about solving this issue? Thank you.
> > > >
> > > > ERROR Unexpected exception, exiting abnormally
> > > > (org.apache.zookeeper.server.ZooKeeperServerMain)
> > > > java.net.BindException: Address already in use
> > > >         at sun.nio.ch.Net.bind0(Native Method)
> > > >         at sun.nio.ch.Net.bind(Net.java:444)
> > > >         at sun.nio.ch.Net.bind(Net.java:436)
> > > >         at
> > > >
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> > > >         at
> > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> > > >         at
> > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:111)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
> > > >
> > > >
> > > >
> > > > On Thu, Jan 29, 2015 at 9:44 AM, Su She <suhshekar52@gmail.com>
> wrote:
> > > >
> > > > > Thank you Guozhang for an updated link.
> > > > >
> > > > > I read the answer for "Why can't my consumers/producers connect to
> > the
> > > > > brokers?". I am confused on a couple things. I can ping the private
> > IP
> > > of
> > > > >  A (Cluster Launcher) from B (EC2 instance). I can't ping the
> public
> > ip
> > > > of
> > > > > A (or google.com) from B which makes sense as B is in a private
> > > subnet.
> > > > > But when I try to connect to the zookeeper at private.ip.A:2181 I
> > get a
> > > > No
> > > > > Route to Host error. Is there a reason why I can ping private.ip.A
> > but
> > > > not
> > > > > connect to private.ip.A:2181?
> > > > >
> > > > > I have not changed any of my server or producer properties. I have
> > > tried
> > > > > to change advertised host name, but no luck.
> > > > >
> > > > > Thanks for the help!
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jan 29, 2015 at 8:12 AM, Guozhang Wang <wangguoz@gmail.com
> >
> > > > wrote:
> > > > >
> > > > >> Sorry my previous link was not complete:
> > > > >>
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers
> > > > >> ?
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, Jan 28, 2015 at 10:56 PM, Su She <suhshekar52@gmail.com>
> > > wrote:
> > > > >>
> > > > >> > Thank you Dillian and Guozhang for the responses.
> > > > >> >
> > > > >> > Yes, Dillian you are understanding my issue correctly. I
am not
> > sure
> > > > >> what
> > > > >> > the best approach to this is...I'm not sure if there's a
way to
> > > > >> whitelist
> > > > >> > certain IPs, create a VPC, use the cluster launcher as the
kafka
> > > > >> > zookeeper/broker. I guess this is more of an AWS question,
but I
> > > > thought
> > > > >> > this is a problem some Kafka users must have solved already.
> > > > >> >
> > > > >> > Edit: I just tried using the cluster launcher as an
> intermediate.
> > I
> > > > >> started
> > > > >> > Zookeeper/Kafka Server on my Cluster launcher and then created
a
> > > > >> > topic/produced messages. I set up a kafka consumer on one
of my
> > > > private
> > > > >> EC2
> > > > >> > instances, but I got a No Route to host error. I pinged
the
> > cluster
> > > > >> > launcher <-> private instance and it works fine. I
was hoping I
> > > could
> > > > >> use
> > > > >> > this is as a temporary solution...any suggestions on this
issue
> > > would
> > > > >> also
> > > > >> > be greatly appreciated. Thanks!
> > > > >> >
> > > > >> > Best,
> > > > >> >
> > > > >> > Su
> > > > >> >
> > > > >> >
> > > > >> > On Wed, Jan 28, 2015 at 9:11 PM, Guozhang Wang <
> > wangguoz@gmail.com>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Su,
> > > > >> > >
> > > > >> > > Does this help for your case?
> > > > >> > >
> > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ
> > > > >> > >
> > > > >> > > Guozhang
> > > > >> > >
> > > > >> > > On Wed, Jan 28, 2015 at 3:36 PM, Dillian Murphey <
> > > > >> > crackshotmule@gmail.com>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Am I understanding your question correctly...
You're asking
> > how
> > > do
> > > > >> you
> > > > >> > > > establish connectivity to an instance in a private
subnet
> from
> > > the
> > > > >> > > outside
> > > > >> > > > world?  Are you thinking in terms of zookeeper
or just
> general
> > > aws
> > > > >> > > network
> > > > >> > > > connectivity?
> > > > >> > > >
> > > > >> > > > On Wed, Jan 28, 2015 at 11:03 AM, Su She <
> > suhshekar52@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > > >
> > > > >> > > > > Hello All,
> > > > >> > > > >
> > > > >> > > > > I have set up a cluster of EC2 instances
using this
> method:
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> http://blogs.aws.amazon.com/bigdata/post/Tx2D0J7QOVRJBRX/Deploying-Cloudera-s-Enterprise-Data-Hub-on-AWS
> > > > >> > > > >
> > > > >> > > > > As you can see the instances are w/in a private
subnet. I
> > was
> > > > >> > wondering
> > > > >> > > > if
> > > > >> > > > > anyone has any advice on how I can set up
a Kafka
> > > > >> zookeeper/server on
> > > > >> > > an
> > > > >> > > > > instance that receives messages from a Kafka
Producer
> > outside
> > > of
> > > > >> the
> > > > >> > > > > private subnet. I have tried using the cluster
launcher,
> > but I
> > > > >> feel
> > > > >> > > like
> > > > >> > > > it
> > > > >> > > > > is not a best practice and only a temporary
situation.
> > > > >> > > > >
> > > > >> > > > > Thank you for the help!
> > > > >> > > > >
> > > > >> > > > > Best,
> > > > >> > > > >
> > > > >> > > > > Su
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > --
> > > > >> > > -- Guozhang
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> -- Guozhang
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message