kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 杨珏吉 <jueji.y...@gmail.com>
Subject kafka 0.9.0 is busy waiting when no message available
Date Thu, 01 Nov 2018 03:38:39 GMT
Hi Kafka experts

I'm using kafka 0.9.0. I made sure that
Fetch.MinBytes(fetch.min.bytes) is set to be 1 and Fetch.MaxWaitTime
(fetch.wait.max.ms) is set to be 250 ms. There are no messages in the
topic but client (I tried golang client `sarama-cluster` and
`bin/kafka-console-consumer.sh`) keeps fetching and broker responses
immediately in 1 ms. This spinning cause huge network and cpu
consumption.

More details:
1. No client side error logs or outputs.
2. For empty topic, the broker will wait 250 ms as expected.
3. After a new message produced and consumed normally, the broker will
be waiting 250 ms as no more message are produced. But once the broker
performs "Deleting index", the broker will again
response immediately with empty message.
the `server.log` at the time when broker change from waiting 250 ms to
responding immediately

[2018-10-26 21:14:52,788] INFO [Group Metadata Manager on Broker 1]:
Removed 0 expired offsets in 0 milliseconds.
(kafka.coordinator.GroupMetadataManager)
[2018-10-26 21:15:21,679] INFO Rolled new log segment for 'sync-0' in
1 ms. (kafka.log.Log)
[2018-10-26 21:15:21,679] INFO Scheduling log segment 0 for log sync-0
for deletion. (kafka.log.Log)
[2018-10-26 21:15:21,680] INFO Rolled new log segment for 'Sync-0' in
0 ms. (kafka.log.Log)
[2018-10-26 21:15:21,680] INFO Scheduling log segment 83 for log
Sync-0 for deletion. (kafka.log.Log)
[2018-10-26 21:16:21,680] INFO Deleting segment 0 from log sync-0.
(kafka.log.Log)
[2018-10-26 21:16:21,680] INFO Deleting index
/data/log/kafka-logs/sync-0/00000000000000000000.index.deleted
(kafka.log.OffsetIndex)
[2018-10-26 21:16:21,680] INFO Deleting segment 83 from log Sync-0.
(kafka.log.Log)
[2018-10-26 21:16:21,680] INFO Deleting index
/data/log/kafka-logs/Sync-0/00000000000000000083.index.deleted
(kafka.log.OffsetIndex)

Does anyone have ever faced this problem?

-- 
Best Regards.

Yang Jueji.

Mime
View raw message