kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Otis Gospodnetic <otis.gospodne...@gmail.com>
Subject Re: Consuming "backwards"?
Date Fri, 06 Dec 2013 23:44:38 GMT
Hi,

On Fri, Dec 6, 2013 at 6:32 PM, Steven Parkes <smparkes@smparkes.net> wrote:

> On Dec 6, 2013, at 2:03 PM, Otis Gospodnetic <otis.gospodnetic@gmail.com>
> wrote:
>
> > but I think the
> > problem is that each time we grab we could get some of the same messages
> we
> > already processed
>
> Doesn't setting the fetchSize to "how far back we need to grab" handle
> that?


I *think* it doesn't, but I'm wrong every day.... N times, so....

I think this is what would happen:
1) imagine 10 messages in the broker m1 - m10
2) consumer grabs last N (=3): m8, m9, m10
3) while it's doing that and before consumer polls for more messages
producer publishes 2 more: m11 and m12
4) consumer now polls again. It asks broker for publisher offset and gets
the answer: 12
5) good, says consumer, let me then fetch everything after offset 12-3=9:
m10, m11, m12

Problem: consumer got m10 again, but it was already processed in 2).

No?  Please correct me if I'm wrong anywhere.

Thanks,
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message