kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmett Butler <emm...@parsely.com>
Subject Re: Ordering in a Kafka Pipe line
Date Fri, 06 Jul 2018 17:38:49 GMT
Kafka provides total ordering only within individual partitions. A topic
with multiple partitions is considered a "partial order", in which multiple
subsets of the topic are considered well-ordered but the topic as a whole
is not. The tradeoff between scalability via partitioning and message
ordering is fundamental to Kafka's design and unfortunately makes it
difficult to built Kafka-based pipelines that rely on strict ordering.

Partition keys are used to ensure that messages sharing a certain
characteristic will always go to the same partition. It's possible that you
could use this to your advantage given the ordering guarantees provided by
individual partitions, but you'd still be limited in your ability to scale
beyond a single partition for any given partition key.

On Fri, Jul 6, 2018 at 10:15 AM chinchu chinchu <chinchutechie@gmail.com>

> Hello,
> We  are building a data pipe line with the following semantics. We need to
> maintain order till the last unit of work is done in this pipe line .We
> cannot have a single partition  since that looses our ability to scale .
> Looked at using partitioning keys ,but does that guarantee order in the
> pipe line as such as opposed to between a topic and consumers?
> produces > consumes enriches and produces >consumes ,does work and produces
> > persistent store
> Thanks,
> Chinchu

Emmett Butler | Senior Software Engineer

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message