kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yongtao You <yongtao_...@yahoo.com.INVALID>
Subject Re: Add Nginx in front of Kafka cluster?
Date Thu, 14 Sep 2017 12:13:09 GMT
I got errors saying the other brokers are not reachable, or something like that. Let me dig
up the exact error messages. I am guessing the problem was that the advertised listeners are
of PLAINTEXT format, but the Nginx requires SSL. But I could be wrong.


    On Thursday, September 14, 2017, 8:07:38 PM GMT+8, Ali Akhtar <ali.rac200@gmail.com>
 How do you know that the brokers don't talk to each other?

On Thu, Sep 14, 2017 at 4:32 PM, Yongtao You <yongtao_you@yahoo.com.invalid>

> Hi,
> I would like to know the right way to setup a Kafka cluster with Nginx in
> front of it as a reverse proxy. Let's say I have 2 Kafka brokers running on
> 2 different hosts; and an Nginx server running on another host. Nginx will
> listen on 2 different ports, and each will forward to one Kafka broker.
> Producers will connect to one of the 2 ports on the Nginx host.
> Nginx-Host: listens on 9000 ssl (forward to <kafka-host-0>:9092 in plain
> text); 9001 ssl (forward to <kafka-host-1>:9092 in plain text);
> Kafka-Host-0: listeners=PLAINTEXT://<kafka-host-0-ip>:9092;
> advertised.listeners=PLAINTEXT://<nginx-host-ip>:9000Kafka-Host-1:
> listeners=PLAINTEXT://<kafka-host-1-ip>:9092; advertised.listeners=
> PLAINTEXT://<nginx-host-ip>:9001
> Ports on Nginx will have SSL enabled so that messages sent from producers
> to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain
> text since it's on the internal network.
> Why have producers go through Nginx? The main reason is that producers
> will only need to open their firewall to a single IP so that even later on
> when I add another Kafka broker, I don't need to modify the firewall of all
> the producers.
> My problem is that I can't make the above setup work. Brokers are unable
> to talk to one another. :(
> So, what's the right way to do this? Anyone has experience setting up
> something similar? Or any recommendations for a different setup that will
> not require changes on the producer's side when new Kafka brokers are added?
> Thanks!Yongtao
> PS. The producers in question are Filebeats (https://www.elastic.co/
> products/beats/filebeat).

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message