kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Indication in FetchResponse that fetch size is too small
Date Thu, 02 Jul 2015 15:32:20 GMT
Hello Apache Kafka community,

Couldn't broker return a special error code in FetchResponse for a given
partition(s) where it detects that there was something to return/read from
partition but actual FetchResponse contains no messages for that partition
since fetch size in FetchRequest for that partition is too small?

I don't see it already in existing Kafka wire protocol docs
<https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol>.
It doesn't seem to be supported, since docs for broker and consumer
configuration mention that the maximums should be in sync, otherwise too
large message could block consumer.

If it was available this feature would be useful for simple consumer API
users, in the cases when range of possible message sizes on same topic is
relatively large (e.g. most very small messages, but some small fraction of
relatively large messages). Would help with being able to  soft and hard
maximums, so most of the time one could read with soft max, and only in
case of this error code is on, attempt a read using hard max limit.

Assuming this feature is not there, I guess what would need to be done at
minimum is:
- minimal change in protocol and docs, just one new error code
- accompanying detection of this condition on broker side, so extend
handling of FetchRequest
- maybe in high level consumer make use of this error code in a special way
(e.g. just log a warning/error that there is potential misconfiguration).

Kind regards,
Stevo Slavic.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message