kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Brown <tombrow...@gmail.com>
Subject Datacenter to datacenter over the open internet
Date Tue, 06 Oct 2015 22:31:53 GMT

How do you consume a kafka topic from a remote location without a dedicated
connection? How do you protect the server?

The setup: data streams into our datacenter. We process it, and publish it
to a kafka cluster. The consumer is located in a different datacenter with
no direct connection. The most efficient scenario would be to setup a
point-to-point link but that idea has no traction with our executives. We
can setup a VPN; While functional, our IT department assures us that it
won't be able to scale.

What we're currently planning is to expose the kafka cluster IP addresses
to the internet, and only allow access via firewall. Each message will be
encrypted with a shared private key, so we're not worried about messages
being intercepted. What we are worried about is this: how brokers refer to
each other-- when a broker directs the consumer to the server that is in
charge of a particular region, does it use the host name (that could be
externally mapped to the public IP) or does it use the detected/private IP

What solution would you use to consume a remote cluster?


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message