kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexei Levashov <alexei.levas...@arrayent.com>
Subject Questions about single consumer per partition approach
Date Wed, 21 Dec 2016 04:36:20 GMT
Hello,
I have a few newbie questions about usage of Kafka as a messaging system.
Kafka version - 0.10.1.0.

1 - Let's assume that I want to ensure time sequence of events i.e. if
message A from producer was published at time t1 to partition P and message
B from the same producer published to partition P at time t2,
I want to consume message A before message B, provided t1<t2.

Question1,
Do I have any choice except one consumer per partition?

2. - If I have one consumer per partition and use
 consumer.assign(partitionList) call to assign consumer to a partition do I
still need group membership for this single consumer?
     I didn't find clear description what is the protocol of interaction
between GroupCoordinator and PartitionLeader
<https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal>
will be in case of  "manual" partition assignment.
     On one hand the API documentation
<https://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html>
says
that :
     "Manual partition assignment does not use group coordination, so
consumer failures will not cause assigned partitions to be rebalanced.
      Each consumer acts independently even if it shares a groupId with
another consumer.
      To avoid offset commit conflicts, you should usually ensure that the
groupId is unique for each consumer instance."

      On the other hand I am still consuming messages in
consumer.poll(timeout) loop and inside this poll() call consumer should
send heartbeats to coordinator.

Question 2 .
     If consumer doesn't send these heartbeats for [*session.timeout.ms
<http://session.timeout.ms>*] period of time  should the partition
ownership be revoked or not?

     If no - does it mean I have to use homegrown heartbeats for consumer
state monitoring? How would the application know that the consumer thread
is dead?
     If yes - what callback to notify the application can I use?
ConsumerRebalanceListener is available only for group subscription.

Thank you.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message