kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sampath kumar <sampath...@gmail.com>
Subject Re: Some of the Kafka consumers are not getting the messages from the broker
Date Wed, 21 Aug 2019 12:36:28 GMT
Lisheng ,

Issue not with  fetch max bytes as same message start processing after
restarting the consumer

Regards,
Sampath

On Wed, Aug 21, 2019 at 4:30 PM Lisheng Wang <wanglisheng81@gmail.com>
wrote:

> Hi Sampath
>
> Can you confirm that "fetch.max.bytes" on consumer is not smaller than
> "message.max.bytes" on broker?
>
> Maybe need you check consumer log to see if can find any clue once you
> enable it. if no any error/exception found on consumer side, maybe need
> change log level to "debug" to get more detail information.
>
> Best,
> Lisheng
>
>
> sampath kumar <sampath088@gmail.com> 于2019年8月21日周三 下午6:30写道:
>
> > Hi Lisheng,
> >
> > Thanks for the response.
> >
> > Right now we have enabled info in the broker However logs not enabled for
> > the consumer client will enable it.
> >
> > Yes, when we manually stop and start the consumer in affected
> microservice
> > instance rebalance triggers and consuming resumes.
> >
> > And in Broker side consumer client status is healthy we verified both in
> > Kafka Manager and Consumer Group Command in Broker CLI, so I guess
> > heartbeat is not the issue and this issue not affected to not complete
> > consumer group only some consumer client in the couple of microservice.
> >
> > Forex: if one of the Service if we have  38 consumer client/thread
> > registered for the consumer group, only 1 client not receiving the
> > messages, rest all getting the messages
> >
> > Anything else you want me to check here?
> >
> > Regards,
> > Sampath
> >
> >
> >
> >
> >
> >
> > On Wed, Aug 21, 2019 at 3:21 PM Lisheng Wang <wanglisheng81@gmail.com>
> > wrote:
> >
> > >  May i know what log level did you configured on consumer and broker?
> > > you say it will resume when rebalance happen, so consumer is alive,
> can
> > > you see any heartbeat information in consumer log?
> > >
> > > Best,
> > > Lisheng
> > >
> > >
> > > sampath kumar <sampath088@gmail.com> 于2019年8月21日周三 下午5:23写道:
> > >
> > > > Hi,
> > > >
> > > > Using Broker 5.3.0, new consumers(Consumers managed by brokers).
> > Brokers
> > > > are deployed in a Kubernetes environment
> > > >
> > > > Number of brokers : 3, Number of 3 Zookeeper setup
> > > >
> > > > One of the Topic "inventory.request" we have 3 replication, with
> insync
> > > > replicas configured as 2 and partition count is 1024
> > > >
> > > > We have 20 instances of microservice subscribe to the above topic,
> each
> > > > instance will have 48 consumers registered as a group
> "agent.group.inv"
> > > >
> > > > Issue :
> > > >
> > > > Here some times a couple of the consumers suddenly stopped receiving
> > the
> > > > request, and lag seems to keep increasing. Only option to recover it
> > > > restart the consumers and invokes rebalancing
> > > >
> > > > ``` agent.group.inv inventory.request 543        17423
>  17612
> > > >       189
> >  agent19.inv.35-6e6eb252-8d26-489b-8d7f-53b25f182f30
> > > >     /10.200.187.103 agent19.inv.35 ```
> > > >
> > > > we checked the thread dump of the consumer, the consumer keeps
> > performing
> > > > polling and assigned with partitions, However not receiving the any
> > > > messages
> > > >
> > > > ``` "inventory.request-agent19.inv.35" #499 prio=1 os_prio=4
> > > > tid=0x00007f88a855b000 nid=0x389 runnable [0x00007f87e8be6000]
> > > >    java.lang.Thread.State: RUNNABLE
> > > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > > >         at
> > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > > >         at
> > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > > >         at
> > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > > >         - locked <0x00000000aa502730> (a sun.nio.ch.Util$3)
> > > >         - locked <0x00000000aa5026b0> (a
> > > > java.util.Collections$UnmodifiableSet)
> > > >         - locked <0x00000000aa502668> (a
> sun.nio.ch.EPollSelectorImpl)
> > > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > > >         at
> > > > org.apache.kafka.common.network.Selector.select(Selector.java:794)
> > > >         at
> > > org.apache.kafka.common.network.Selector.poll(Selector.java:467)
> > > >         at
> > > > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159)
> > > > ```
> > > >
> > > >
> > > > No errors are observed consumer client, brokers and also resource
> issue
> > > not
> > > > seen.
> > > >
> > > > Can you please help us in identifying the root cause for this
> consumer
> > > > client behavior?
> > > >
> > > > Please let me know if any other details required?
> > > >
> > > > --
> > > > Regards,
> > > > Sampath
> > > >
> > >
> >
> >
> > --
> > Regards,
> > Sampath
> >
>


-- 
Regards,
Sampath

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message