kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: understanding OffsetOutOfRangeException's....
Date Mon, 23 Dec 2013 23:46:26 GMT
Jason,

You can search the correlation id in the public access log on the servers
to get the consumer information.

As for logging, I agree that we should use the same level on both sides.
Could you file a jira for this?

Guozhang


On Mon, Dec 23, 2013 at 3:09 PM, Jason Rosenberg <jbr@squareup.com> wrote:

> In our broker logs, we occasionally see errors like this:
>
> 2013-12-23 05:02:08,456 ERROR [kafka-request-handler-2] server.KafkaApis -
> [KafkaApi-45] Error when processing fetch request for partition [mytopic,0]
> offset 204243601 from consumer with correlation id 130341
> kafka.common.OffsetOutOfRangeException: Request for offset 204243601 but we
> only have log segments in the range 204343397 to 207423640.
>
> I assume this means there's a consumer that has fallen behind consuming
> messages, and the log retention policy has removed messages before they
> could be consumed by the consumer.
>
> However, I'm not 100% which consumer it is, and it looks like the only info
> we have is the correlation id of the consumer, e.g.:
>
> "from consumer with correlation id 130341"
>
> Is there a way to know which consumer this refers to?  It seems there are
> far more correlation id's than there are consumers.  Would it be possible
> to provide a bit more descriptive error message here, so we can immediately
> know which consumer is falling behind?
>
> We do see a corresponding entry in the consumer logs too:
>
> 2013-12-23 05:02:08,797  WARN
> [ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45]
> consumer.ConsumerFetcherThread -
> [ConsumerFetcherThread-myconsumergroup-1387353494862-7aa0c61d-0-45],
> Current offset 204243601 for partition [mytopic,0] out of range; reset
> offset to 204343397
>
> But it would be nice to be able to also use the broker log to quickly find
> consumers with issues.
>
> Also, I'm not sure, why is logging the event as an ERROR in the broker, but
> a WARN in the consumer?
>
> Jason
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message