kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Best practices
Date Mon, 21 May 2018 16:18:52 GMT
If you specify some bootstrap.servers, after connecting the producer
will learn about all available brokers automatically, by fetching
cluster metadata from the first broker it connects. Thus, in practice it
is usually sufficient to specify 3 to 5 brokers (in case one is down,
the producer can "bootstrap" itself connecting to any other broker in
the list initially).

Also note, that producer do no write to an arbitrary broker: for each
partition, there is a dedicates leader and the producer sends all write
to the leader (as explained in the blog post is shared in my last reply
--- please read it:
https://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/)

For best practices, it's recommended to configure producer retries via
config parameter `retries` (default is zero). For this case, if a write
fails, the producer will retry the write internally (potentially to a
different broker in case the leader changed). Only after all retries are
exhausted, you would get a callback that indicates the failed write.


-Matthias

On 5/21/18 5:40 AM, Pavel Sapozhnikov wrote:
> If a process failed to write a message into one broker what is the best
> practice to write to same topic on different broker? Is there one? I should
> be able to get a list of brokers programmatically from zk path /brokers/ids
> ?
> 
> On Sun, May 20, 2018, 3:21 PM Matthias J. Sax <matthias@confluent.io> wrote:
> 
>> You can register a callback for each sent record to learn about
>> successful write or fail:
>>
>>> producer.send(record, callback);
>>
>> For replication, you don't need to send twice. If the replication factor
>> is configured broker side, the broker take care of replication
>> automatically.
>>
>> You can also configure when you want to be informed about successful
>> write: before or after replication.
>>
>> This blog post should help:
>>
>> https://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/
>>
>>
>>
>> -Matthias
>>
>> On 5/20/18 11:00 AM, Pavel Sapozhnikov wrote:
>>> Hello
>>>
>>> What are the best practices when it comes to publishing a message into
>>> kafka. When sending a message into Kafka is it possible to know if that
>>> message has successfully been published? If not, what is the best
>> practice
>>> to know when messages are not getting published?
>>>
>>> Second question.
>>>
>>> If I have two kafka brokers and very simplistic one kafka topic
>> replicated
>>> on both. Do I need to send to both brokers. What are the best practices
>> for
>>> that.
>>>
>>
>>
> 


Mime
View raw message