kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor Kravzov <igork.ine...@gmail.com>
Subject Topics, partitions and keys
Date Sat, 28 May 2016 15:14:09 GMT
I need some clarification on subject.
In Kafka documentations I found the following:

Kafka only provides a total order over messages *within* a partition, not
between different partitions in a topic. Per-partition ordering combined
with the ability to partition data by key is sufficient for most
applications. However, if you require a total order over messages this can
be achieved with a topic that has only one partition, though this will mean
only one consumer process per consumer group.

So here are my questions:
1. Does it mean if i want to have more than 1 consumer (from the same
group) reading from one topic I need to have more than 1 partition?

2. Does it mean I need same amount of partitions as amount of consumers for
the same group?

3. How many consumers can read from one partition?

Also have some questions regarding relationship between keys and partitions
with regard to API. I only looked at .net APIs (especially one from MS)
 but looks like the mimic Java API.
I see when using a producer to send a message to a topic there is a key
parameter. But when consumer reads from a topic there is a partition number.

1. How are partitions numbered? Starting from 0 or 1?
2. What exactly relationship between a key and partition?
As I understand some function on key will determine a partition. is that
correct?
3. If I have 2 partitions in a topic and want some particular messages go
to one partition and other messages go to another I should use a specific
key for one specific partition, and the rest for another?
4. What if I have 3 partitions and one type of messages to one particular
partition and the rest to other 2?
5. How in general I send messages to a particular partition in order to
know  for a consumer from where to read?
Or I better off with multiple topics?

Thanks in advance.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message