kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bo Xu <box...@gmail.com>
Subject Re: messages intermittently dropped in a single partition configuration
Date Thu, 28 Apr 2016 21:28:30 GMT
Ok, I think I found the cause of the problem. The default value
of max_in_flight_requests_per_connection is 5 in Python producer. This
turns out too small for my environment and application. When this value is
reached and the producer tries several times and still fails, the message
is dropped. And since send() is asynchronous, there is no way for the
producer to report the failure immediately. So one solution is to block for
'synchronous' sends as described at
http://kafka-python.readthedocs.io/en/1.0.0/usage.html. But this slows down
the producing speed. Another solution, which I end up with, is to
significantly increase max_in_flight_requests_per_connection, e.g., to 1024.



On Thu, Apr 28, 2016 at 10:31 AM, Bo Xu <boxu08@gmail.com> wrote:

> PS: The message dropping occurred intermittently, not all at the end. For
> example, it is the 10th, 15th, 18th messages that are missing. It it were
> all at the end, it would be understandable because I'm not using flush() to
> force transmitting.
>
> Bo
>
>
> On Thu, Apr 28, 2016 at 10:15 AM, Bo Xu <boxu08@gmail.com> wrote:
>
>> I set up a simple Kafka configuration, with one topic and one partition.
>> I have a Python producer to continuously publish messages to the Kafka
>> server and a Python consumer to receive messages from the server. Each
>> message is about 10K bytes, far smaller than
>> socket.request.max.bytes=104857600. What I found is that the consumer
>> intermittently missed some messages. I checked the server log file and
>> found that these messages are missing there as well. So it looks like these
>> message were never stored by the server. I also made sure that the producer
>> did not receive any error for every message that it published (using
>> send()).
>>
>> Any clues what could have caused the problem?
>>
>> Thanks,
>> Bo
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message