kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sekhar2, Sinu" <Sinu_Sekh...@intuit.com>
Subject Kafka Producers and Virtual Kafka Endpoint
Date Wed, 21 Jan 2015 18:44:15 GMT
Hello Kafka Users

I am new to Kafka and am evaluating it for using it for our high throughput messaging needs.

I tested my producers using two different approaches.
(a) metadata.broker.list=kafka1:9092,kafka2:9092,kafka3:9092
And
(b) metadata.broker.list=vip:9092
Where vip is an ELB (elastic load balancer in AWS) pointing to kafka1, kafka2 and kafka3

partitions=3,replication=3 for the topic that I am using.

Both the approaches seem to work from a producer standpoint and I am able to consume these
messages fine as well.

My goal is to be able to hide the underlying hosts from the producer and be able to use a
virtual endpoint (like the vip).
But I am not sure how the clients will know which node is hosting a specific partition. If
my number of partitions are less than the number of nodes, then there is a chance that the
request can end up in a node that is not hosting the partition as master….what happens in
that case?

Has anyone implemented Kafka with a virtual endpoint using ELB, or some other load balancer
like Nginx in AWS?
What are the recommendations to hide the underlying node information from the kafka clients
when it comes to producers OR even consumers?

Any guidance will be extremely helpful!

Cheers!
Sinu
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message