kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philip O'Toole <phi...@loggly.com>
Subject Re: message order, guarenteed?
Date Fri, 14 Jun 2013 15:02:02 GMT
Another idea. If a set of messages arrive over a single TCP connection, route to a partition
depending on TCP connection.

To be honest, these approaches, while they work, may not scale when the message rate is high.
If at all possible, try to think of a way to remove this requirement from your design. For
example, a design might have a sequence number assigned to each message before it goes into
Kafka (a time-based UUID, for example), and something later in the pipe line sorts it all
out. Kafka then does what is does best, IMHO, a high-performance reliable, message bus. 


On Jun 14, 2013, at 7:37 AM, David Arthur <mumrah@gmail.com> wrote:

> Simple example of how to take advantage of this behavior:
> Suppose you're sending document updates through Kafka. If you use the document ID as
the message key and the default hash partitioner, the updates for a given document will exist
on the same partition and come into the consumer in order.
> On 6/10/13 8:37 AM, Neha Narkhede wrote:
>> Kafka guarantees order per topic  partition per source client.
>> Thanks,
>> Neha
>> On Jun 9, 2013 5:33 PM, "S Ahmed" <sahmed1020@gmail.com> wrote:
>>> I understand that there are no guarantees per say that a message may be a
>>> duplicate (its the consumer's job to guarantee that), but when it comes to
>>> message order, is kafka built in such a way that it is impossible to get
>>> messages in the wrong order?
>>> Certain use cases might not be sensitive to order, but when order is very
>>> important, is kafka the wrong tool for the job or is there a way to get
>>> this requirement?

View raw message