kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Chen <tnac...@gmail.com>
Subject Re: Data loss detection
Date Tue, 03 Jun 2014 22:00:16 GMT
Hi Maung,

If your required.acks is 1 then the producer only ensures that one
broker receives the data before it's sucessfully returned to the
client.

Therefore if the broker crashes and lost all the data then you lose
data, or similarly it can happen even before the data is fsynced.

To ensure there are more copies of your data in case of failure
scenarios you want to increase your required.acks to more than 1 to
tolerate failuries.

Also async producer doesn't wait until the data is sent before it
returns, as it buffers and writes asynchronously. To ensure each write
that has a succesful response is written you want to use the sync
producer.

Tim

On Tue, Jun 3, 2014 at 2:13 PM, Maung Than <maung_than@apple.com> wrote:
> Hi,
>
> We are seeing less data on the brokers than we send form the producers:  84 GB to 58
GB.
>
> What is the best way to ensure / detect if all data has been send properly to the brokers
from the producers.
>
> Is there any logs that we can check on the producers?
>
> Configuration is 5 Brokers, 2 producers, no replication factor, async and ask is 1 and
no compression.
>
> Thanks,
> Maung

Mime
View raw message