kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From A A <ek.tha....@hotmail.com>
Subject Dropping messages ?
Date Mon, 10 Feb 2014 09:21:46 GMT

We have been using Kafka(0.8) for the past few months with the following setup
Kafka Broker - 1Zookeepers Ensemble - 3Partitions per topic - 3

Yesterday, while running Stress tests in one of the QA machines , we observed that a few messages
which were produced within a couple of milliseconds of each other did not reach the Kafka
consumer. ie There was no trace of that message at the consumer end. 

We decided to check whether we had any errors at our side or there was a network issue. We
did not find any issue. We then decided to check whether we can find that message in one of
the Kafka partitions. The message was found in one of the topic partitions. 

We are not sure why Kafka did not notify any consumers about the message. Are there any special
cases where Kafka silently drops a message ? 

We also found a delay in the notifications/watches triggered from zookeeper. We are not sure
whether these are related ? It will be difficult to reproduce as the test probably took a
few days to complete. But surely we did lose approximately 5% of the messages. We have logs
of messages being produced at the producer side and corresponding entries in Kafka partitions
logs. But nothing at the consumer side. The only repeating pattern was that the messages were
probably produced within the same millisecond. So if you have a sequence of messages which
was produced in the same millisecond like M0, M1, M2, M3 ie 4 messages. We probably have M0,M1,M3
but not M2. This is puzzling as to how only message is dropped out of the given 4. 

We use the High Level Kafka Producer and Consumer. Both are single threaded(at our end).

Does kafka need its own dedicated zookeeper ensemble ? We also use the same zookeeper ensemble
as  our configuration service. 

Unfortunately, we did not have DEBUG messages at the server enabled during the setup. Although,
NO error messages were observed during the same time period. 

Before we try running the same Tests again, can someone please shed more light as to the reasons
why kafka dropped a few messages ?

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message