kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Schuhegger <Christian.Schuheg...@gmx.de>
Subject Kafka and no guarantee that every published message is actually received by the broker
Date Tue, 04 Feb 2014 16:52:41 GMT
Hello all,

I was reading in the following paper:

http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf

the following paragraph:

-- snip start --
There are a few reasons why Kafka performed much better. First,
the Kafka producer currently doesn’t wait for acknowledgements
from the broker and sends messages as faster as the broker can
handle. This significantly increased the throughput of the
publisher. With a batch size of 50, a single Kafka producer almost
saturated the 1Gb link between the producer and the broker. This
is a valid optimization for the log aggregation case, as data must
be sent asynchronously to avoid introducing any latency into the
live serving of traffic. We note that without acknowledging the
producer, there is no guarantee that every published message is
actually received by the broker. For many types of log data, it is
desirable to trade durability for throughput, as long as the number
of dropped messages is relatively small. However, we do plan to
address the durability issue for more critical data in the future.
-- snip end --

And I was wondering if this is still true or if the plans for the future 
as described above to address the durability issue for more critical 
data were realized?

Many thanks,
-- 
Christian Schuhegger



Mime
View raw message