kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Loddengaard <a...@confluent.io>
Subject Re: Topics, partitions and keys
Date Tue, 31 May 2016 18:37:42 GMT
Hi Igor, see inline:

On Sat, May 28, 2016 at 8:14 AM, Igor Kravzov <igork.inexso@gmail.com>
wrote:

> I need some clarification on subject.
> In Kafka documentations I found the following:
>
> Kafka only provides a total order over messages *within* a partition, not
> between different partitions in a topic. Per-partition ordering combined
> with the ability to partition data by key is sufficient for most
> applications. However, if you require a total order over messages this can
> be achieved with a topic that has only one partition, though this will mean
> only one consumer process per consumer group.
>
> So here are my questions:
> 1. Does it mean if i want to have more than 1 consumer (from the same
> group) reading from one topic I need to have more than 1 partition?
>

Yes.


>
> 2. Does it mean I need same amount of partitions as amount of consumers for
> the same group?
>

No. If you have more partitions than consumers, consumers will consume from
more than one topic.


>
> 3. How many consumers can read from one partition?
>

Only one.


>
> Also have some questions regarding relationship between keys and partitions
> with regard to API. I only looked at .net APIs (especially one from MS)
>  but looks like the mimic Java API.
> I see when using a producer to send a message to a topic there is a key
> parameter. But when consumer reads from a topic there is a partition
> number.
>
> 1. How are partitions numbered? Starting from 0 or 1?
> 2. What exactly relationship between a key and partition?
>

Using the default partitioner, and assuming you don't add new partitions,
all messages with the same key are guaranteed to land in the same partition.


> As I understand some function on key will determine a partition. is that
> correct?
> 3. If I have 2 partitions in a topic and want some particular messages go
> to one partition and other messages go to another I should use a specific
> key for one specific partition, and the rest for another?
>

If you always want one key to go to a specific partition, and another key
to go to a different partition, you can use a custom partitioner or the
Java API:

http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html


> 4. What if I have 3 partitions and one type of messages to one particular
> partition and the rest to other 2?
>

Same as above. Although, in most cases, all messages in a topic are the
same "type" of message. A topic is like a database table in this way.


> 5. How in general I send messages to a particular partition in order to
> know  for a consumer from where to read?
>

See above.


> Or I better off with multiple topics?
>

This will depend on your application.


>
> Thanks in advance.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message