kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: Indication in FetchResponse that fetch size is too small
Date Fri, 03 Jul 2015 08:48:59 GMT
Thanks for heads up Joel!

Kind regards,
Stevo Slavic.

On Thu, Jul 2, 2015 at 10:17 PM, Joel Koshy <jjkoshy.w@gmail.com> wrote:

> A somewhat related request came up in another thread and I think it is
> reasonable to provide this. However, there are already some indicators
> that you can use:
> - The consumer iterator throws a MessageSizeTooLargeException if it
>   cannot extract any messages out of the next chunk.
> - If you are using the simple consumer, the fetch response includes
>   the high watermark - so if the HW > your fetch offset but there are
>   no iterable messages, then you will know that your fetch size needs
>   to increase.
>
> Thanks,
>
> Joel
>
> On Thu, Jul 02, 2015 at 05:32:20PM +0200, Stevo Slavić wrote:
> > Hello Apache Kafka community,
> >
> > Couldn't broker return a special error code in FetchResponse for a given
> > partition(s) where it detects that there was something to return/read
> from
> > partition but actual FetchResponse contains no messages for that
> partition
> > since fetch size in FetchRequest for that partition is too small?
> >
> > I don't see it already in existing Kafka wire protocol docs
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
> >.
> > It doesn't seem to be supported, since docs for broker and consumer
> > configuration mention that the maximums should be in sync, otherwise too
> > large message could block consumer.
> >
> > If it was available this feature would be useful for simple consumer API
> > users, in the cases when range of possible message sizes on same topic is
> > relatively large (e.g. most very small messages, but some small fraction
> of
> > relatively large messages). Would help with being able to  soft and hard
> > maximums, so most of the time one could read with soft max, and only in
> > case of this error code is on, attempt a read using hard max limit.
> >
> > Assuming this feature is not there, I guess what would need to be done at
> > minimum is:
> > - minimal change in protocol and docs, just one new error code
> > - accompanying detection of this condition on broker side, so extend
> > handling of FetchRequest
> > - maybe in high level consumer make use of this error code in a special
> way
> > (e.g. just log a warning/error that there is potential misconfiguration).
> >
> > Kind regards,
> > Stevo Slavic.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message