Hi,
On Fri, Dec 6, 2013 at 6:32 PM, Steven Parkes <smparkes@smparkes.net> wrote:
> On Dec 6, 2013, at 2:03 PM, Otis Gospodnetic <otis.gospodnetic@gmail.com>
> wrote:
>
> > but I think the
> > problem is that each time we grab we could get some of the same messages
> we
> > already processed
>
> Doesn't setting the fetchSize to "how far back we need to grab" handle
> that?
I *think* it doesn't, but I'm wrong every day.... N times, so....
I think this is what would happen:
1) imagine 10 messages in the broker m1 - m10
2) consumer grabs last N (=3): m8, m9, m10
3) while it's doing that and before consumer polls for more messages
producer publishes 2 more: m11 and m12
4) consumer now polls again. It asks broker for publisher offset and gets
the answer: 12
5) good, says consumer, let me then fetch everything after offset 12-3=9:
m10, m11, m12
Problem: consumer got m10 again, but it was already processed in 2).
No? Please correct me if I'm wrong anywhere.
Thanks,
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
|