kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lisheng Wang <wanglishen...@gmail.com>
Subject Re: Some of the Kafka consumers are not getting the messages from the broker
Date Wed, 21 Aug 2019 22:44:15 GMT
Hi Sampath

Maybe need to check log to see if can find any clue, hard to say why those
happened.

Best,
Lisheng


sampath kumar <sampath088@gmail.com> 于2019年8月22日周四 上午2:24写道:

> Hi Lisheng,
>
> I guess the issue is not with message.max.bytes, same messages consumer
> after just running rebalancing.
>
> Regards,
> Sampath
>
> On Wed, Aug 21, 2019 at 7:57 PM Lisheng Wang <wanglisheng81@gmail.com>
> wrote:
>
> > Hi Sampath
> >
> > the description of fetch.max.bytes is following from
> > https://kafka.apache.org/documentation/#consumerconfigs
> >
> > The maximum amount of data the server should return for a fetch request.
> > Records are fetched in batches by the consumer, and if the first record
> > batch in the first non-empty partition of the fetch is larger than this
> > value, the record batch will still be returned to ensure that the
> consumer
> > can make progress. As such, this is not a absolute maximum. The maximum
> > record batch size accepted by the broker is defined via
> > message.max.bytes (broker
> > config) or max.message.bytes (topic config). Note that the consumer
> > performs multiple fetches in parallel.
> >
> > the point is "if the first record batch in the first non-empty partition
> of
> > the fetch is larger than this value, the record batch will still be
> > returned to ensure that the consumer can make progress".
> >
> > just wanna make sure the issue you are facing is not related to that.
> >
> >
> > Best,
> > Lisheng
> >
> >
> > sampath kumar <sampath088@gmail.com> 于2019年8月21日周三 下午8:36写道:
> >
> > > Lisheng ,
> > >
> > > Issue not with  fetch max bytes as same message start processing after
> > > restarting the consumer
> > >
> > > Regards,
> > > Sampath
> > >
> > > On Wed, Aug 21, 2019 at 4:30 PM Lisheng Wang <wanglisheng81@gmail.com>
> > > wrote:
> > >
> > > > Hi Sampath
> > > >
> > > > Can you confirm that "fetch.max.bytes" on consumer is not smaller
> than
> > > > "message.max.bytes" on broker?
> > > >
> > > > Maybe need you check consumer log to see if can find any clue once
> you
> > > > enable it. if no any error/exception found on consumer side, maybe
> need
> > > > change log level to "debug" to get more detail information.
> > > >
> > > > Best,
> > > > Lisheng
> > > >
> > > >
> > > > sampath kumar <sampath088@gmail.com> 于2019年8月21日周三 下午6:30写道:
> > > >
> > > > > Hi Lisheng,
> > > > >
> > > > > Thanks for the response.
> > > > >
> > > > > Right now we have enabled info in the broker However logs not
> enabled
> > > for
> > > > > the consumer client will enable it.
> > > > >
> > > > > Yes, when we manually stop and start the consumer in affected
> > > > microservice
> > > > > instance rebalance triggers and consuming resumes.
> > > > >
> > > > > And in Broker side consumer client status is healthy we verified
> both
> > > in
> > > > > Kafka Manager and Consumer Group Command in Broker CLI, so I guess
> > > > > heartbeat is not the issue and this issue not affected to not
> > complete
> > > > > consumer group only some consumer client in the couple of
> > microservice.
> > > > >
> > > > > Forex: if one of the Service if we have  38 consumer client/thread
> > > > > registered for the consumer group, only 1 client not receiving the
> > > > > messages, rest all getting the messages
> > > > >
> > > > > Anything else you want me to check here?
> > > > >
> > > > > Regards,
> > > > > Sampath
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Aug 21, 2019 at 3:21 PM Lisheng Wang <
> > wanglisheng81@gmail.com>
> > > > > wrote:
> > > > >
> > > > > >  May i know what log level did you configured on consumer and
> > broker?
> > > > > > you say it will resume when rebalance happen, so consumer is
> alive,
> > > > can
> > > > > > you see any heartbeat information in consumer log?
> > > > > >
> > > > > > Best,
> > > > > > Lisheng
> > > > > >
> > > > > >
> > > > > > sampath kumar <sampath088@gmail.com> 于2019年8月21日周三
下午5:23写道:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Using Broker 5.3.0, new consumers(Consumers managed by
> brokers).
> > > > > Brokers
> > > > > > > are deployed in a Kubernetes environment
> > > > > > >
> > > > > > > Number of brokers : 3, Number of 3 Zookeeper setup
> > > > > > >
> > > > > > > One of the Topic "inventory.request" we have 3 replication,
> with
> > > > insync
> > > > > > > replicas configured as 2 and partition count is 1024
> > > > > > >
> > > > > > > We have 20 instances of microservice subscribe to the above
> > topic,
> > > > each
> > > > > > > instance will have 48 consumers registered as a group
> > > > "agent.group.inv"
> > > > > > >
> > > > > > > Issue :
> > > > > > >
> > > > > > > Here some times a couple of the consumers suddenly stopped
> > > receiving
> > > > > the
> > > > > > > request, and lag seems to keep increasing. Only option
to
> recover
> > > it
> > > > > > > restart the consumers and invokes rebalancing
> > > > > > >
> > > > > > > ``` agent.group.inv inventory.request 543        17423
> > > >  17612
> > > > > > >       189
> > > > >  agent19.inv.35-6e6eb252-8d26-489b-8d7f-53b25f182f30
> > > > > > >     /10.200.187.103 agent19.inv.35 ```
> > > > > > >
> > > > > > > we checked the thread dump of the consumer, the consumer
keeps
> > > > > performing
> > > > > > > polling and assigned with partitions, However not receiving
the
> > any
> > > > > > > messages
> > > > > > >
> > > > > > > ``` "inventory.request-agent19.inv.35" #499 prio=1 os_prio=4
> > > > > > > tid=0x00007f88a855b000 nid=0x389 runnable [0x00007f87e8be6000]
> > > > > > >    java.lang.Thread.State: RUNNABLE
> > > > > > >         at sun.nio.ch.EPollArrayWrapper.epollWait(Native
> Method)
> > > > > > >         at
> > > > > sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> > > > > > >         at
> > > > > > sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
> > > > > > >         at
> > > > > sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
> > > > > > >         - locked <0x00000000aa502730> (a sun.nio.ch.Util$3)
> > > > > > >         - locked <0x00000000aa5026b0> (a
> > > > > > > java.util.Collections$UnmodifiableSet)
> > > > > > >         - locked <0x00000000aa502668> (a
> > > > sun.nio.ch.EPollSelectorImpl)
> > > > > > >         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
> > > > > > >         at
> > > > > > >
> > org.apache.kafka.common.network.Selector.select(Selector.java:794)
> > > > > > >         at
> > > > > > org.apache.kafka.common.network.Selector.poll(Selector.java:467)
> > > > > > >         at
> > > > > > >
> > org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:539)
> > > > > > >         at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
> > > > > > >         at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> > > > > > >         at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281)
> > > > > > >         at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225)
> > > > > > >         at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1159)
> > > > > > > ```
> > > > > > >
> > > > > > >
> > > > > > > No errors are observed consumer client, brokers and also
> resource
> > > > issue
> > > > > > not
> > > > > > > seen.
> > > > > > >
> > > > > > > Can you please help us in identifying the root cause for
this
> > > > consumer
> > > > > > > client behavior?
> > > > > > >
> > > > > > > Please let me know if any other details required?
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > Sampath
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > Sampath
> > > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > Sampath
> > >
> >
>
>
> --
> Regards,
> Sampath
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message