kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: How to fetch old messages from kafka
Date Fri, 06 Feb 2015 03:03:50 GMT
> We can reset the offset and get first 10 messages, but since we need to back in reverse
sequence, suppose user has consumed messages upto 100 offset , currently there are only last
10 messages are visible, from 100 -90, now I want to retrieve messages from 80 to 90, how
can we do that?

I'm not sure I fully understand the question, but you can always
rewind to an explicit offset - i.e., if you know that you want offset
80 onwards, then set your fetch offset to 80 and start fetching from
there until you hit offset 90 (or whatever limit you need). This
assumes that those messages are still present though on the broker.

On Wed, Feb 04, 2015 at 02:59:17PM +0530, Snehalata Nagaje wrote:
> 
> 
> Hi Mayuresh,
> 
> 
> Thanks for quick response.
> 
> We can reset the offset and get first 10 messages, but since we need to back in reverse
sequence, suppose user has consumed messages upto 100 offset , currently there are only last
10 messages are visible, from 100 -90, now I want to retrieve messages from 80 to 90, how
can we do that?
> 
> Can we use getOffsetBefore() function to get valid offset before given time, this will
return all valid offsets. we can get all valid offsets before latest time.
> 
> Then we can fetch messages from any given valid offset returned from getOffsetBefore().
> 
> is this correct approach?
> 
> Thanks,
> Snehalata
> 
> ----- Original Message -----
> From: gharatmayuresh15@gmail.com
> To: users@kafka.apache.org
> Sent: Wednesday, February 4, 2015 11:24:46 AM
> Subject: Re: How to fetch old messages from kafka
> 
> In that case you will have to maintain the offsets consumed and reset the offsets in
case you need to consume from past.
> 
> For example, suppose you have a userA for which you have a partitionA for topic TopicA.
Each page shown to user increments the offset by 10. You have consumed till offset 100 and
the user wants to go back 1 page you will have to reset the offset for TopicA partitionA in
the zookeeper. Since you are using simple consumer the offset management has to be done by
your application.
> 
> Thanks,
> 
> Mayuresh
> 
> Sent from my iPhone
> 
> > On Feb 3, 2015, at 9:18 PM, Snehalata Nagaje <snehalata.nagaje@harbingergroup.com>
wrote:
> > 
> > 
> > 
> > Hi ,
> > 
> > 
> > We are using kafka for storing messages in chat application.
> > 
> > Currently we divided each topic in multiple partitions. each partition stores data
for given customer who uses the application.
> > 
> > Right now on very first request, application fetches log from kafka from earliest
valid offset to maxiumum 100000 bytes. hence it reads all messages for given topic
> > 
> > for given partition. Now we want to apply pagination as linkedin, facebook does.
Only latest 10-15 messages should be displayed. And then on scroll down
> > 
> > fetch next set of previous messages, we are using Simple consumer to fetch messages.
> > 
> > Can you please guide on this?
> > 
> > 
> > Thanks,
> > Snehalata


Mime
View raw message