kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "YuanJia Li" <yuanjia8...@163.com>
Subject Re: Re: Pagecache cause OffsetOutOfRangeException
Date Wed, 03 Dec 2014 02:30:41 GMT
Hi Guozhang,
    My kafka works in product environment, a large messages are produced or consumed. So it
is not easy to get the accurate offset through the GetOffset tool when an OffsetOutOfRangeException
happens.But in my application, I have coded comparing the consuming offset with the latest
offset got by SimpleConsumer.getOffsetsBefore API.
    I don't quite understand the kernel pagecache, so I wonder the delay between writting
and reading.

Thanks,
Yuanjia





From: Guozhang Wang
Date: 2014-12-03 03:23
To: users@kafka.apache.org
Subject: Re: Pagecache cause OffsetOutOfRangeException
Yuanjia,

I am not sure that pagecache can be the cause of this, could you attach
your full stack trace and use the GetOffset tool Manikumar mentioned to
make sure the offset does exist in the broker?

Guozhang

On Tue, Dec 2, 2014 at 7:50 AM, Manikumar Reddy <kumar@nmsworks.co.in>
wrote:

> You can check the latest/earliest offsets of a given topic by running
> GetOffsetShell.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-GetOffsetShell
>
> On Tue, Dec 2, 2014 at 2:05 PM, yuanjia8947 <yuanjia8947@163.com> wrote:
>
> > Hi all,
> > I'm using kafka 0.8.0 release now. And I often encounter the problem
> > OffsetOutOfRangeException when cosuming message by simple consumer API.
> > But I'm sure that the consuming offset is smaller than the latest offset
> > got from OffsetRequest.
> > Can it be caused by that new messages are wrote to kernel's pagecache and
> > not flush to the file yet,
> > while I'm consuming new messages from the file?
> > How fix it?
> >
> > Thanks,
> > liyuanjia
> >
> >
> >
> >
> >
> > liyuanjia
>



-- 
-- Guozhang
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message