kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Lost of messages at C++ Kafka client
Date Wed, 07 Aug 2013 15:09:05 GMT
Yes, in 0.7, one of the limitations is that the producer doesn't wait for
any acknowledgement from the broker. So, if the broker is down, messages
still in the client socket buffer may be lost. In 0.8, we added the option
so that the producer can wait for an ack, which will help this particular
kind of data loss.

Thanks,

Jun


On Wed, Aug 7, 2013 at 2:38 AM, Tianning Zhang <Tianning.Zhang@zanox.com>wrote:

> Hi,
>
> I am using Kafka (version 0.7.2) for publishing events from some C++
> applications (ca. 20K event /sec). The C++ client is used, which is based
> on asio.
> With the asynchronous and batched messaging, one issue I have is that in
> case Kafka server is broken, the client will not notice the socket lost
> immediately.  Messages go lost between the moment the socket broke and the
> moment when the C++ client issues an error notification.  Message lost is
> not tolerated in our application. Although we have some proprietary
> component to deal with the problem – it is not a convenient solution.
>
> As I understand, this is a known issue. But I would like to know more
> about experiences and best practices for dealing this problems. What can
> future versions of Kafka offer to deal with this problems?
>
> Kind regards
>
> Tianning Zhang ::  tianning.zhang@zanox.com<mailto:
> tianning.zhang@zanox.com>
>
>
>
> [disclaimer logo: ZANOX.de AG]
>
> STRAIGHT TO PERFORMANCE
>
> --------------------------------------------------------------------------------
>
> ZANOX.de AG | Headquarters: Berlin AG Charlottenburg | HRB 75459 | VAT
> identification number: DE 209981705
> Executive Board: Thomas Joosten (CEO) | Stefanie Lüdecke (CSO) | Joachim
> Piroth (CFO) | Christian Rebernik (CTO)
> Chairman of the Supervisory Board: Ralph Büchi
>
> This e-mail and any attachments may contain confidential and/or privileged
> information.
> If you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and delete this e-mail from
> your system.
> Any other use, copying, disclosure or distribution is strictly forbidden.
>
> [100 Partnerprogramme 2012]
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message