kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Stopford <...@confluent.io>
Subject Re: Questions about single consumer per partition approach
Date Wed, 21 Dec 2016 14:06:51 GMT
Hi Alexi

Typically you would use a key to guarantee that messages with the same key
have a global ordering, rather than using manual assignment. Kafka will
send all messages with the same key to the same partition. If you need
global ordering, spanning all messages from a single producer, you can use
a single partition topic. This will limit you to one active consumer per
consumer group as the consumer group protocol guarantees that a partition
can only be assigned to one consumer, within a group, at one time.


On Wed, Dec 21, 2016 at 4:36 AM Alexei Levashov <
alexei.levashov@arrayent.com> wrote:

> Hello,
> I have a few newbie questions about usage of Kafka as a messaging system.
> Kafka version -
> 1 - Let's assume that I want to ensure time sequence of events i.e. if
> message A from producer was published at time t1 to partition P and message
> B from the same producer published to partition P at time t2,
> I want to consume message A before message B, provided t1<t2.
> Question1,
> Do I have any choice except one consumer per partition?
> 2. - If I have one consumer per partition and use
>  consumer.assign(partitionList) call to assign consumer to a partition do I
> still need group membership for this single consumer?
>      I didn't find clear description what is the protocol of interaction
> between GroupCoordinator and PartitionLeader
> <
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal
> >
> will be in case of  "manual" partition assignment.
>      On one hand the API documentation
> <
> https://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
> >
> says
> that :
>      "Manual partition assignment does not use group coordination, so
> consumer failures will not cause assigned partitions to be rebalanced.
>       Each consumer acts independently even if it shares a groupId with
> another consumer.
>       To avoid offset commit conflicts, you should usually ensure that the
> groupId is unique for each consumer instance."
>       On the other hand I am still consuming messages in
> consumer.poll(timeout) loop and inside this poll() call consumer should
> send heartbeats to coordinator.
> Question 2 .
>      If consumer doesn't send these heartbeats for [*session.timeout.ms
> <http://session.timeout.ms>*] period of time  should the partition
> ownership be revoked or not?
>      If no - does it mean I have to use homegrown heartbeats for consumer
> state monitoring? How would the application know that the consumer thread
> is dead?
>      If yes - what callback to notify the application can I use?
> ConsumerRebalanceListener is available only for group subscription.
> Thank you.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message