kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <joe.st...@stealth.ly>
Subject Re: Consuming "backwards"?
Date Fri, 06 Dec 2013 15:26:55 GMT
The fetch requests are very flexible to-do what you want with them.

Take a look at SimpleConsumerShell.scala as a reference

You could pass in OffsetRequest.LatestTime (-1) with a fetch size of 3 and
then just keep doing that over and over again.

I think that will do exactly what you are looking to-do.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/


On Fri, Dec 6, 2013 at 10:04 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi,
>
> On Fri, Dec 6, 2013 at 9:38 AM, Tom Brown <tombrown52@gmail.com> wrote:
>
> > Do you mean you want to start from the most recent data and go backwards
> to
> > the oldest data, or that you want to start with old data and consume
> > forwards?
> >
>
> Forwards is the "normal way".  I'm looking for the "abnormal way", of
> course ;) i.e. backwards.
> If the following are the messages that came in, oldest to newest:
>
> M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12
>
> Then I'd love to be able to consume from the end, say in batches of 3, like
> this:
>
> get last 3: M10 M11 M12
> get last 3: M7 M8 M9
> get last 3: M4 M5 M6
> get last 3: M1 M2 M3
>
> Of course, if messages keep coming in, then the new ones that arrive would
> get picked up first and, eventually, assuming Consumer can consume faster
> than messages are produced, all messages will get consumed.
>
> But the important/key part is that any new ones that arrive will get picked
> up first.
>
> If the former, it would be difficult or impossible in 0.7.x, but I think
> > doable in 0.8.x. (They added some sort of message index). If the latter,
> > that is easily accomplished in both versions.
> >
>
> I'd love to know if that's really so and how to do it!
>
> We are looking to move to Kafka 0.8 in January and to add performance
> monitoring for Kafka 0.8 to SPM (see
>
> http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/
> )
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
> > On Friday, December 6, 2013, Otis Gospodnetic wrote:
> >
> > > Hi,
> > >
> > > Does Kafka offer a way to consume messages in batches, but "from the
> > end"?
> > >
> > > This would be valuable to have in all systems where the most recent
> data
> > is
> > > a lot more important than older data, such as performance metrics, and
> > > maybe even logs....maybe also trading/financial data, and such.
> > >
> > > Any chance of this sort of functionality ever making it into Kafka, or
> is
> > > this simply not implementable due to some underlying assumptions, or
> data
> > > structures, or ... ?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Performance Monitoring * Log Analytics * Search Analytics
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message