kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Chen <tnac...@gmail.com>
Subject Re: Data loss detection
Date Wed, 04 Jun 2014 04:58:52 GMT
By the way if you're using async producer how do you verify that you
sent all the data from the producer?

Do you shutdown the producer before you check?

Tim

On Tue, Jun 3, 2014 at 3:27 PM, Maung Than <maung_than@apple.com> wrote:
> Thanks, Tim.
>
> We are just trying to benchmark the kafka producers and there is no issue with cluster
or brokers being down in this case.
>
> We are seeing way less data on the borers after calculating the sizes of the logs on
the brokers) and there is no compression.
>
> We send 84 GB, but total logs sizes are only 58 GB on the brokers.
>
> Since replication factor is zero, can we use ack other than 1?
>
> Maung
>
> On Jun 3, 2014, at 3:00 PM, Timothy Chen <tnachen@gmail.com> wrote:
>
>> Hi Maung,
>>
>> If your required.acks is 1 then the producer only ensures that one
>> broker receives the data before it's sucessfully returned to the
>> client.
>>
>> Therefore if the broker crashes and lost all the data then you lose
>> data, or similarly it can happen even before the data is fsynced.
>>
>> To ensure there are more copies of your data in case of failure
>> scenarios you want to increase your required.acks to more than 1 to
>> tolerate failuries.
>>
>> Also async producer doesn't wait until the data is sent before it
>> returns, as it buffers and writes asynchronously. To ensure each write
>> that has a succesful response is written you want to use the sync
>> producer.
>>
>> Tim
>>
>> On Tue, Jun 3, 2014 at 2:13 PM, Maung Than <maung_than@apple.com> wrote:
>>> Hi,
>>>
>>> We are seeing less data on the brokers than we send form the producers:  84 GB
to 58 GB.
>>>
>>> What is the best way to ensure / detect if all data has been send properly to
the brokers from the producers.
>>>
>>> Is there any logs that we can check on the producers?
>>>
>>> Configuration is 5 Brokers, 2 producers, no replication factor, async and ask
is 1 and no compression.
>>>
>>> Thanks,
>>> Maung
>

Mime
View raw message