kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sampath kumar <sampath...@gmail.com>
Subject Some of the Kafka consumers are not getting the messages from the broker
Date Wed, 21 Aug 2019 09:23:15 GMT
Hi,

Using Broker 5.3.0, new consumers(Consumers managed by brokers). Brokers
are deployed in a Kubernetes environment

Number of brokers : 3, Number of 3 Zookeeper setup

One of the Topic "inventory.request" we have 3 replication, with insync
replicas configured as 2 and partition count is 1024

We have 20 instances of microservice subscribe to the above topic, each
instance will have 48 consumers registered as a group "agent.group.inv"

Issue :

Here some times a couple of the consumers suddenly stopped receiving the
request, and lag seems to keep increasing. Only option to recover it
restart the consumers and invokes rebalancing

``` agent.group.inv inventory.request 543        17423           17612
      189             agent19.inv.35-6e6eb252-8d26-489b-8d7f-53b25f182f30
    /10.200.187.103 agent19.inv.35 ```

we checked the thread dump of the consumer, the consumer keeps performing
polling and assigned with partitions, However not receiving the any messages

``` "inventory.request-agent19.inv.35" #499 prio=1 os_prio=4
tid=0x00007f88a855b000 nid=0x389 runnable [0x00007f87e8be6000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00000000aa502730> (a sun.nio.ch.Util$3)
        - locked <0x00000000aa5026b0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00000000aa502668> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at
org.apache.kafka.common.network.Selector.select(Selector.java:794)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
        at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539)
        at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
        at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
        at
org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281)
        at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225)
        at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159)
```


No errors are observed consumer client, brokers and also resource issue not
seen.

Can you please help us in identifying the root cause for this consumer
client behavior?

Please let me know if any other details required?

-- 
Regards,
Sampath

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message