kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Jacot <dja...@confluent.io>
Subject Re: Kafka cluster metadata update question
Date Wed, 21 Aug 2019 07:23:34 GMT
Hello Emanuel,

It does not work because Kafka advertises the IP it is configured with (the
ones of the broker) in the metadata response. I think you have two options:

1. You configure the default listener of each broker to advertise the IP
address of their proxy instance. It means that the inter-broker traffic
will also go trough the proxies; or

2. You keep the default listener with the internal IP and add another one
in the config which advertises the external IP (the proxy one).

There is a good explanation available here:
https://rmoff.net/2018/08/02/kafka-listeners-explained/

Best,
David

On Mon, Aug 12, 2019 at 5:32 PM Emanuel Muhr <emanuel.muhr@mhp.com> wrote:

> Hi,
>
> we are using Kafka in an environment where we have restricted access to
> the Kafka brokers.
> The access needs to happen via TCP proxies. So, in my setup I have 3 Kafka
> brokers (broker1-3) for which I created 3 proxy instances and I have set
> these proxy instances as the bootstrap servers:
> bootstrap.servers = [ssl://proxy1:9092, ssl://proxy2:9092,
> ssl://proxy3:9092]
>
> Upon startup of a Kafka client I see that there is a successful metadata
> update and the client receive the original kafka broker names:
> [2019-08-12 16:09:56,921] DEBUG Updated cluster metadata version 2 to
> Cluster(id = lFFAv0okS12Vg2rvVbsKdg, nodes = [broker1:9092 (id: 1 rack:
> r1), broker3:9092 (id: 3 rack: r3), broker2:9092 (id: 2 rack: r2)],
> partitions = [Partition(topic = mytopic, partition = 0, leader = 2,
> replicas = [2], isr = [2], offlineReplicas = [])], controller =
> broker1:9092 (id: 1 rack: r1)) (org.apache.kafka.clients.Metadata)
>
> So, the initial connection via the proxy instances does work correctly.
> The Kafka documentation states that the bootstrap.servers are only used for
> bootstrapping so I assume that if the hostnames do not match the ones from
> the brokers they will be replaced.
> Additionally I found this C++ library documentation (
> https://github.com/edenhill/librdkafka/wiki/FAQ#number-of-broker-tcp-connections)
> which states the following:
> The initial bootstrap broker connections will only be used for Metadata
> queries, unless the hostname and port of a bootstrap broker exactly matches
> the hostname and port of a broker returned in the Metadata response (this
> is the advertised.listeners broker configuration property), in which case
> the bootstrap broker connection will be associated with that broker's
> broker id and used for the full protocol set (such as producing or
> consuming).
>
> We do not use this library but Spring Kafka Java library instead. However,
> the documentation makes sense with what I'm observing.
>
> I'm asking if there is any way to disable this behavior and let Kafka use
> the hostnames from the bootstrap broker list for producing and consuming as
> well.
> I do not see any other way so that we can use Kafka in this restricted
> scenario unless we introduce some DNS rerouting.
>
> I would appreciate any insights on this topic if I'm missing information
> or if there is another way to achieve the successful connection under these
> circumstances.
>
> Thanks
> Emanuel
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message