kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ali Akhtar <ali.rac...@gmail.com>
Subject Re: Add Nginx in front of Kafka cluster?
Date Thu, 14 Sep 2017 12:31:24 GMT
If you ssh to the server where you got this error, are you able to ping the
ip of node 7 on the port its trying to reach?

On Thu, Sep 14, 2017 at 5:20 PM, Yongtao You <yongtao_you@yahoo.com> wrote:

> I'm getting a lot of these in the server.log:
>
>
> [2017-09-14 20:18:32,753] WARN Connection to node 7 could not be
> established. Broker may not be available. (org.apache.kafka.clients.
> NetworkClient)
>
>
> where node 7 is another broker in the cluster.
>
>
> Thanks.
>
> -Yongtao
>
>
> On Thursday, September 14, 2017, 8:13:09 PM GMT+8, Yongtao You <
> yongtao_you@yahoo.com> wrote:
>
>
> I got errors saying the other brokers are not reachable, or something like
> that. Let me dig up the exact error messages. I am guessing the problem was
> that the advertised listeners are of PLAINTEXT format, but the Nginx
> requires SSL. But I could be wrong.
>
>
> Thanks!
>
> -Yongtao
>
>
> On Thursday, September 14, 2017, 8:07:38 PM GMT+8, Ali Akhtar <
> ali.rac200@gmail.com> wrote:
>
>
> How do you know that the brokers don't talk to each other?
>
> On Thu, Sep 14, 2017 at 4:32 PM, Yongtao You <
> yongtao_you@yahoo.com.invalid>
> wrote:
>
> > Hi,
> > I would like to know the right way to setup a Kafka cluster with Nginx in
> > front of it as a reverse proxy. Let's say I have 2 Kafka brokers running
> on
> > 2 different hosts; and an Nginx server running on another host. Nginx
> will
> > listen on 2 different ports, and each will forward to one Kafka broker.
> > Producers will connect to one of the 2 ports on the Nginx host.
> > Nginx-Host: listens on 9000 ssl (forward to <kafka-host-0>:9092 in plain
> > text); 9001 ssl (forward to <kafka-host-1>:9092 in plain text);
> >
> > Kafka-Host-0: listeners=PLAINTEXT://<kafka-host-0-ip>:9092;
> > advertised.listeners=PLAINTEXT://<nginx-host-ip>:9000Kafka-Host-1:
> > listeners=PLAINTEXT://<kafka-host-1-ip>:9092; advertised.listeners=
> > PLAINTEXT://<nginx-host-ip>:9001
> > Ports on Nginx will have SSL enabled so that messages sent from producers
> > to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain
> > text since it's on the internal network.
> > Why have producers go through Nginx? The main reason is that producers
> > will only need to open their firewall to a single IP so that even later
> on
> > when I add another Kafka broker, I don't need to modify the firewall of
> all
> > the producers.
> > My problem is that I can't make the above setup work. Brokers are unable
> > to talk to one another. :(
> > So, what's the right way to do this? Anyone has experience setting up
> > something similar? Or any recommendations for a different setup that will
> > not require changes on the producer's side when new Kafka brokers are
> added?
> >
> > Thanks!Yongtao
> > PS. The producers in question are Filebeats (https://www.elastic.co/
> > products/beats/filebeat).
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message