kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: Consuming "backwards"?
Date Fri, 06 Dec 2013 15:04:54 GMT
Hi,

On Fri, Dec 6, 2013 at 9:38 AM, Tom Brown <tombrown52@gmail.com> wrote:

> Do you mean you want to start from the most recent data and go backwards to
> the oldest data, or that you want to start with old data and consume
> forwards?
>

Forwards is the "normal way".  I'm looking for the "abnormal way", of
course ;) i.e. backwards.
If the following are the messages that came in, oldest to newest:

M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12

Then I'd love to be able to consume from the end, say in batches of 3, like
this:

get last 3: M10 M11 M12
get last 3: M7 M8 M9
get last 3: M4 M5 M6
get last 3: M1 M2 M3

Of course, if messages keep coming in, then the new ones that arrive would
get picked up first and, eventually, assuming Consumer can consume faster
than messages are produced, all messages will get consumed.

But the important/key part is that any new ones that arrive will get picked
up first.

If the former, it would be difficult or impossible in 0.7.x, but I think
> doable in 0.8.x. (They added some sort of message index). If the latter,
> that is easily accomplished in both versions.
>

I'd love to know if that's really so and how to do it!

We are looking to move to Kafka 0.8 in January and to add performance
monitoring for Kafka 0.8 to SPM (see
http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/)

Thanks,
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/



> On Friday, December 6, 2013, Otis Gospodnetic wrote:
>
> > Hi,
> >
> > Does Kafka offer a way to consume messages in batches, but "from the
> end"?
> >
> > This would be valuable to have in all systems where the most recent data
> is
> > a lot more important than older data, such as performance metrics, and
> > maybe even logs....maybe also trading/financial data, and such.
> >
> > Any chance of this sort of functionality ever making it into Kafka, or is
> > this simply not implementable due to some underlying assumptions, or data
> > structures, or ... ?
> >
> > Thanks,
> > Otis
> > --
> > Performance Monitoring * Log Analytics * Search Analytics
> > Solr & Elasticsearch Support * http://sematext.com/
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message