kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: Which node do you send data to?
Date Sat, 07 Mar 2015 06:27:48 GMT

Kafka (and it's clients) handle failover automatically for you. When you
create a topic, you can select a replication factor. For a replication
factor n, each partition of the topic will be replicated to n different
brokers. At any given time, one of those brokers is considered the leader
for that topic and that is the only server you can communicate with to
produce new messages. That broker will then make sure the data is copied to
the other replicas. If that leader fails, one of the replicas will take
over and the producer will have to send data to that node.

But all of this should happen automatically. As long as you set the
bootstrap.servers setting (for the new producer) so that at least one of
them is always available (ideally it should just include all the brokers),
then you shouldn't have to worry about this. Of course if a node fails you
have to deal with bringing it back up/moving it to a new machine, but
producers should continue to function normally by moving traffic to the new

On Fri, Mar 6, 2015 at 9:05 PM, Daniel Moreno <dan@max2.com> wrote:

> Hi Spencer,
> You can configure your producers with a list of brokers. You can add all,
> but usually at least two of the brokers in your cluster.
> Kind Regards,
> Daniel Moreno
> On Mar 6, 2015, at 23:43, Spencer Owen <sowen@netdocuments.com<mailto:
> sowen@netdocuments.com>> wrote:
> I've setup a kafka cluster with 3 nodes.
> Which node should I push the data to? I would normally push to kafka01,
> but if that node goes down, then the entire cluster goes down.
> How have other people solved this. Maybe a nginx reverse proxy?
> http://stackoverflow.com/questions/28911410/which-node-should-i-push-data-to-in-a-cluster
> ____________________________________
> This message may contain information that is privileged or confidential.
> If you received this transmission in error, please notify the sender by
> reply e-mail and delete the message and any attachments.
> Warning: All email sent to this address will be received by the
> NetDocuments corporate e-mail system and is subject to archival and review
> by someone other than the recipient.
> The services to which this email (or any email in reply or to which this
> email replies) relates are provided solely by NetDocuments and NetDocuments
> has no responsibility for the services to which this email relates (or any
> email in reply or to which this email replies).


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message