kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lisheng Wang <wanglishen...@gmail.com>
Subject Re: Some of the Kafka consumers are not getting the messages from the broker
Date Wed, 21 Aug 2019 10:59:52 GMT
Hi Sampath

Can you confirm that "fetch.max.bytes" on consumer is not smaller than
"message.max.bytes" on broker?

Maybe need you check consumer log to see if can find any clue once you
enable it. if no any error/exception found on consumer side, maybe need
change log level to "debug" to get more detail information.

Best,
Lisheng


sampath kumar <sampath088@gmail.com> 于2019年8月21日周三 下午6:30写道:

> Hi Lisheng,
>
> Thanks for the response.
>
> Right now we have enabled info in the broker However logs not enabled for
> the consumer client will enable it.
>
> Yes, when we manually stop and start the consumer in affected microservice
> instance rebalance triggers and consuming resumes.
>
> And in Broker side consumer client status is healthy we verified both in
> Kafka Manager and Consumer Group Command in Broker CLI, so I guess
> heartbeat is not the issue and this issue not affected to not complete
> consumer group only some consumer client in the couple of microservice.
>
> Forex: if one of the Service if we have  38 consumer client/thread
> registered for the consumer group, only 1 client not receiving the
> messages, rest all getting the messages
>
> Anything else you want me to check here?
>
> Regards,
> Sampath
>
>
>
>
>
>
> On Wed, Aug 21, 2019 at 3:21 PM Lisheng Wang <wanglisheng81@gmail.com>
> wrote:
>
> >  May i know what log level did you configured on consumer and broker?
> > you say it will resume when rebalance happen, so consumer is alive,  can
> > you see any heartbeat information in consumer log?
> >
> > Best,
> > Lisheng
> >
> >
> > sampath kumar <sampath088@gmail.com> 于2019年8月21日周三 下午5:23写道:
> >
> > > Hi,
> > >
> > > Using Broker 5.3.0, new consumers(Consumers managed by brokers).
> Brokers
> > > are deployed in a Kubernetes environment
> > >
> > > Number of brokers : 3, Number of 3 Zookeeper setup
> > >
> > > One of the Topic "inventory.request" we have 3 replication, with insync
> > > replicas configured as 2 and partition count is 1024
> > >
> > > We have 20 instances of microservice subscribe to the above topic, each
> > > instance will have 48 consumers registered as a group "agent.group.inv"
> > >
> > > Issue :
> > >
> > > Here some times a couple of the consumers suddenly stopped receiving
> the
> > > request, and lag seems to keep increasing. Only option to recover it
> > > restart the consumers and invokes rebalancing
> > >
> > > ``` agent.group.inv inventory.request 543        17423           17612
> > >       189
>  agent19.inv.35-6e6eb252-8d26-489b-8d7f-53b25f182f30
> > >     /10.200.187.103 agent19.inv.35 ```
> > >
> > > we checked the thread dump of the consumer, the consumer keeps
> performing
> > > polling and assigned with partitions, However not receiving the any
> > > messages
> > >
> > > ``` "inventory.request-agent19.inv.35" #499 prio=1 os_prio=4
> > > tid=0x00007f88a855b000 nid=0x389 runnable [0x00007f87e8be6000]
> > >    java.lang.Thread.State: RUNNABLE
> > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > >         at
> sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > >         at
> > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > >         at
> sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > >         - locked <0x00000000aa502730> (a sun.nio.ch.Util$3)
> > >         - locked <0x00000000aa5026b0> (a
> > > java.util.Collections$UnmodifiableSet)
> > >         - locked <0x00000000aa502668> (a sun.nio.ch.EPollSelectorImpl)
> > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > >         at
> > > org.apache.kafka.common.network.Selector.select(Selector.java:794)
> > >         at
> > org.apache.kafka.common.network.Selector.poll(Selector.java:467)
> > >         at
> > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539)
> > >         at
> > >
> > >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
> > >         at
> > >
> > >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> > >         at
> > >
> > >
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281)
> > >         at
> > >
> > >
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225)
> > >         at
> > >
> > >
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159)
> > > ```
> > >
> > >
> > > No errors are observed consumer client, brokers and also resource issue
> > not
> > > seen.
> > >
> > > Can you please help us in identifying the root cause for this consumer
> > > client behavior?
> > >
> > > Please let me know if any other details required?
> > >
> > > --
> > > Regards,
> > > Sampath
> > >
> >
>
>
> --
> Regards,
> Sampath
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message