kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: Indication in FetchResponse that fetch size is too small
Date Thu, 02 Jul 2015 20:17:40 GMT
A somewhat related request came up in another thread and I think it is
reasonable to provide this. However, there are already some indicators
that you can use:
- The consumer iterator throws a MessageSizeTooLargeException if it
  cannot extract any messages out of the next chunk.
- If you are using the simple consumer, the fetch response includes
  the high watermark - so if the HW > your fetch offset but there are
  no iterable messages, then you will know that your fetch size needs
  to increase.



On Thu, Jul 02, 2015 at 05:32:20PM +0200, Stevo Slavić wrote:
> Hello Apache Kafka community,
> Couldn't broker return a special error code in FetchResponse for a given
> partition(s) where it detects that there was something to return/read from
> partition but actual FetchResponse contains no messages for that partition
> since fetch size in FetchRequest for that partition is too small?
> I don't see it already in existing Kafka wire protocol docs
> <https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol>.
> It doesn't seem to be supported, since docs for broker and consumer
> configuration mention that the maximums should be in sync, otherwise too
> large message could block consumer.
> If it was available this feature would be useful for simple consumer API
> users, in the cases when range of possible message sizes on same topic is
> relatively large (e.g. most very small messages, but some small fraction of
> relatively large messages). Would help with being able to  soft and hard
> maximums, so most of the time one could read with soft max, and only in
> case of this error code is on, attempt a read using hard max limit.
> Assuming this feature is not there, I guess what would need to be done at
> minimum is:
> - minimal change in protocol and docs, just one new error code
> - accompanying detection of this condition on broker side, so extend
> handling of FetchRequest
> - maybe in high level consumer make use of this error code in a special way
> (e.g. just log a warning/error that there is potential misconfiguration).
> Kind regards,
> Stevo Slavic.

View raw message