kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philip O'Toole" <phi...@loggly.com>
Subject Re: Lost of messages at C++ Kafka client
Date Wed, 07 Aug 2013 15:05:47 GMT
If I understand what you are asking, I have dealt successfully with the
same type of issue. It can take more than one Boost async_write() over a
broken connection before the client software notices that the connection is

The best way to detect if a connection is broken is not by detecting that
the ASIO async_write() call returns an error (or raises an exception).
Instead, also start an async_read on the socket (doesn't matter if you
never expect to actually read from it). This call will return immediately
when a connection is broken, allowing much quicker detection of this event.
See if this helps.


On Wed, Aug 7, 2013 at 2:38 AM, Tianning Zhang <Tianning.Zhang@zanox.com>wrote:

> Hi,
> I am using Kafka (version 0.7.2) for publishing events from some C++
> applications (ca. 20K event /sec). The C++ client is used, which is based
> on asio.
> With the asynchronous and batched messaging, one issue I have is that in
> case Kafka server is broken, the client will not notice the socket lost
> immediately.  Messages go lost between the moment the socket broke and the
> moment when the C++ client issues an error notification.  Message lost is
> not tolerated in our application. Although we have some proprietary
> component to deal with the problem – it is not a convenient solution.
> As I understand, this is a known issue. But I would like to know more
> about experiences and best practices for dealing this problems. What can
> future versions of Kafka offer to deal with this problems?
> Kind regards
> Tianning Zhang ::  tianning.zhang@zanox.com<mailto:
> tianning.zhang@zanox.com>
> [disclaimer logo: ZANOX.de AG]
> --------------------------------------------------------------------------------
> ZANOX.de AG | Headquarters: Berlin AG Charlottenburg | HRB 75459 | VAT
> identification number: DE 209981705
> Executive Board: Thomas Joosten (CEO) | Stefanie Lüdecke (CSO) | Joachim
> Piroth (CFO) | Christian Rebernik (CTO)
> Chairman of the Supervisory Board: Ralph Büchi
> This e-mail and any attachments may contain confidential and/or privileged
> information.
> If you are not the intended recipient (or have received this e-mail in
> error) please notify the sender immediately and delete this e-mail from
> your system.
> Any other use, copying, disclosure or distribution is strictly forbidden.
> [100 Partnerprogramme 2012]

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message