kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sabarish Sasidharan <sabarish....@gmail.com>
Subject Re: Out of order message processing with Kafka Streams
Date Sat, 18 Mar 2017 18:36:54 GMT

What you state would work for aggregations, but not for state machines and


On 19 Mar 2017 12:01 a.m., "Hans Jespersen" <hans@confluent.io> wrote:

> The only way to make sure A is consumed first would be to delay the
> consumption of message B for at least 15 minutes which would fly in the
> face of the principals of a true streaming platform so the short answer to
> your question is "no" because that would be batch processing not stream
> processing.
> However, Kafka Streams does handle late arriving data. So if you had some
> analytics that computes results on a time window or a session window then
> Kafka streams will compute on the stream in real time (processing message
> B) and then later when message A arrives it will put that message back into
> the right temporal context and publish an amended result for the proper
> time/session window as if message B were consumed in the timestamp order
> before message A. The end result of this flow is that you eventually get
> the same results you would get in a batch processing system but with the
> added benefit of getting intermediary result at much lower latency.
> -hans
> /**
>  * Hans Jespersen, Principal Systems Engineer, Confluent Inc.
>  * hans@confluent.io (650)924-2670
>  */
> On Sat, Mar 18, 2017 at 10:29 AM, Ali Akhtar <ali.rac200@gmail.com> wrote:
> > Is it possible to have Kafka Streams order messages correctly by their
> > timestamps, even if they arrived out of order?
> >
> > E.g, say Message A with a timestamp of 5:00 PM and Message B with a
> > timestamp of 5:15 PM, are sent.
> >
> > Message B arrives sooner than Message A, due to network issues.
> >
> > Is it possible to make sure that, across all consumers of Kafka Streams
> > (even if they are across different servers, but have the same consumer
> > group), Message A is consumed first, before Message B?
> >
> > Thanks.
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message