kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hans Jespersen <h...@confluent.io>
Subject Re: Can anyone help me to send messages in their original order?
Date Sat, 26 May 2018 20:26:19 GMT
There are two concepts in Kafka that are not always familiar to people who have used other
pub/sub systems. 

1) partitions: 

Kafka topics are partitioned which means a single topic is sharded into multiple pieces that
are distributed across multiple brokers in the cluster for parallel processing.

Order is guaranteed per partition (not per topic).

You can think of each kafka topic partition like an exclusive queue is traditional messaging
systems and order is not guaranteed when the data is spread out across multiple queues in
tradition messaging either.

2) keys

Kafka messages have keys in addition the value (I.e body) and the header. When messages are
published with the same key they will be all be sent in order to the same partition.

If messages are published with a “null” key then they will be spread out round robin across
all partitions (which is what you have done).


Conclusion 

You will see ordered delivery if your either use a key when you publish or create a topic
with one partition.


-hans

> On May 26, 2018, at 7:59 AM, Raymond Xie <xie3208080@gmail.com> wrote:
> 
> Thanks. By default, can you explain me why I received the message in wrong order? Note
there are only 9 lines from 1 to 9, but on consumer side their original order becomes messed
up.
> 
> ~~~sent from my cell phone, sorry if there is any typo
> 
> Hans Jespersen <hans@confluent.io> 于 2018年5月26日周六 上午12:16写道:
>> If you create a topic with one partition they will be in order.
>> 
>> Alternatively if you publish with the same key for every message they will be in
the same order even if your topic has more than 1 partition.
>> 
>> Either way above will work for Kafka.
>> 
>> -hans
>> 
>> > On May 25, 2018, at 8:56 PM, Raymond Xie <xie3208080@gmail.com> wrote:
>> > 
>> > Hello,
>> > 
>> > I just started learning Kafka and have the environment setup on my
>> > hortonworks sandbox at home vmware.
>> > 
>> > test.csv is what I want the producer to send out:
>> > 
>> > more test1.csv ./kafka-console-producer.sh --broker-list
>> > sandbox.hortonworks.com:6667 --topic kafka-topic2
>> > 
>> > 1, abc
>> > 2, def
>> > ...
>> > 8, vwx
>> > 9, zzz
>> > 
>> > What I received are all the content of test.csv, however, not in their
>> > original order;
>> > 
>> > kafka-console-consumer.sh --zookeeper 192.168.112.129:2181 --topic
>> > kafka-topic2
>> > 
>> > 2, def
>> > 1, abc
>> > ...
>> > 9, zzz
>> > 8, vwx
>> > 
>> > 
>> > I read from google that partition could be the feasible solution, however,
>> > my questions are:
>> > 
>> > 1. for small files like this one, shall I really do the partitioning? how
>> > small a partition would be acceptable to ensure the sequence?
>> > 2. for big files, each partition could still contain multiple lines, how to
>> > ensure all the lines in each partition won't get messed up on consumer side?
>> > 
>> > 
>> > I also want to know what is the best practice to process large volume of
>> > data through kafka? There should be better way other than console command.
>> > 
>> > Thank you very much.
>> > 
>> > 
>> > 
>> > *------------------------------------------------*
>> > *Sincerely yours,*
>> > 
>> > 
>> > *Raymond*

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message