kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Zhou <frank.z...@itiviti.com>
Subject Re: Consumer hangs at poll() and never throw exception on client
Date Mon, 16 Dec 2019 03:43:21 GMT
Hi Matthias,

Seems the issue is related to transaction. Let me expand a bit on our flow
of this. Our application will use single consumer to fetch the records,
deal with the data and then multiple producers to send data afterwards.
There could be multiple applications sending data to a single topic as
well. So the above issue happens when transactions are enabled when
multiple applications are running(Transaction ids are different among
producers and applications). Today I tested with transaction disabled, and
everything seems fine. Do you see any issue for our usage on transactions?

On Tue, Dec 10, 2019 at 4:42 PM Matthias J. Sax <matthias@confluent.io>
wrote:

> Hard to say why the second consumer does not fetch data.
>
> I would try to inspect the logs at DEBUG level to get more insight what
> going on.
>
>
> -Matthias
>
> On 12/10/19 12:27 AM, Frank Zhou wrote:
> > "earliest",  as we need everything from the start
> >
> > On Tue, Dec 10, 2019 at 4:22 PM Matthias J. Sax <matthias@confluent.io>
> > wrote:
> >
> >> What is your `auto.offset.reset` policy?
> >>
> >> On 12/10/19 12:02 AM, Frank Zhou wrote:
> >>> Hi,
> >>>
> >>> Thanks for your prompt reply. Consumers have different group.id in our
> >>> case, as we just need all the data in the topic, so every time the
> >>> application starts, the consumer will be in a new group. For current
> >>> testing environment, we just one partition for one topic. though we
> have
> >>> tried with multiple partitions, and situation happened as well.
> >>>
> >>> On Tue, Dec 10, 2019 at 3:56 PM Matthias J. Sax <matthias@confluent.io
> >
> >>> wrote:
> >>>
> >>>> Are both consumers in the same consumer group, ie, use the same `
> >> group.id
> >>>> `?
> >>>>
> >>>> If yes, how many partitions does the topic have? If it has only one
> >>>> partition, the observed behavior is expected, because a single
> >>>> partitions can only be read by a single consumer instance per consumer
> >>>> group. The second consumer won't have any assigned partitions to read
> >> from.
> >>>>
> >>>> -Matthias
> >>>>
> >>>> On 12/9/19 11:12 PM, Frank Zhou wrote:
> >>>>> Hi,
> >>>>>
> >>>>> I am quite new to Kafka, and I have encountered a weird case during
> QA
> >>>>> stage for our application. Now we have 2 consumers consuming same
> topic
> >>>>> in kafka cluster. The first started consumer works fine and get
> closed
> >>>>> after getting all the messages. After that, the second one started
> and
> >>>>> just hang there in the poll(), which we set the Duration to 10
> seconds,
> >>>>> and will quit the loop if no records are fetched. From kafka server
> >> log,
> >>>>> I don't see warning or error for the second consumer.
> >>>>>
> >>>>> Attached client/server log related to both consumers. I am using
> 2.3.1
> >>>>> client, 2.2.2 kafka servers. Looking forward to you feedback.
> >>>>>
> >>>>>
> >>>>> Best regards,
> >>>>> --
> >>>>> *Frank Zhou*
> >>>>> R&D, Itiviti
> >>>>> Java Developer
> >>>>> D +852 2521 7480
> >>>>> frank.zhou@itiviti.com <mailto:frank.zhou@itiviti.com>
> >>>>>
> >>>>> ______________________________
> >>>>>
> >>>>> itiviti.com <https://www.itiviti.com/>
> >>>>>
> >>>>> /The information contained in or attached to this email is strictly
> >>>>> confidential. If you are not the intended recipient, please notify
us
> >>>>> immediately by telephone and return the message to us./
> >>>>>
> >>>>> /Email communications by definition contain personal information.
The
> >>>>> Itiviti group of companies is subject to European data protection
> >>>>> regulations. Itiviti’s Privacy Notice is available at
> www.itiviti.com
> >>>>> <http://www.itiviti.com/>. Itiviti expects the recipient of
this
> email
> >>>>> to be compliant with Itiviti’s Privacy Notice and applicable
> >>>>> regulations. Please advise us immediately at
> >>>>> dataprotectionteam@Itiviti.com if you are not compliant with these./
> >>>>>
> >>>>>
> >>>>> ______________________________
> >>>>>
> >>>>> itiviti.com <https://www.itiviti.com/>
> >>>>> <https://www.linkedin.com/company/itiviti> ^Follow Itiviti
on
> Linkedin
> >>>>> <https://www.linkedin.com/company/10438325/>
> >>>>>
> >>>>> The information contained in or attached to this email is strictly
> >>>>> confidential. If you are not the intended recipient, please notify
us
> >>>>> immediately by telephone and return the message to us. Email
> >>>>> communications by definition contain personal information. The
> Itiviti
> >>>>> group of companies is subject to European data protection
> regulations.
> >>>>>
> >>>>> Itiviti’s Privacy Notice is available at www.itiviti.com
> >>>>> <http://www.itiviti.com/>. Itiviti expects the recipient of
this
> email
> >>>>> to be compliant with Itiviti’s Privacy Notice and applicable
> >>>>> regulations. Please advise us immediately at
> >>>>> dataprotectionteam@Itiviti.com if you are not compliant with these.
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>
>

-- 
*Frank Zhou*
R&D, Itiviti
Java Developer
D +852 2521 7480
frank.zhou@itiviti.com

______________________________

itiviti.com <https://www.itiviti.com/>

*The information contained in or attached to this email is strictly
confidential. If you are not the intended recipient, please notify us
immediately by telephone and return the message to us.*

*Email communications by definition contain personal information. The
Itiviti group of companies is subject to European data protection
regulations. Itiviti’s Privacy Notice is available at www.itiviti.com
<http://www.itiviti.com/>. Itiviti expects the recipient of this email to
be compliant with Itiviti’s Privacy Notice and applicable regulations.
Please advise us immediately at dataprotectionteam@Itiviti.com if you are
not compliant with these.*

-- 
______________________________

itiviti.com <https://www.itiviti.com/>
 
<https://www.linkedin.com/company/itiviti> Follow Itiviti on Linkedin 
<https://www.linkedin.com/company/10438325/>




The information contained 
in or attached to this email is strictly confidential. If you are not the 
intended recipient, please notify us immediately by telephone and return 
the message to us. Email communications by definition contain personal 
information. The Itiviti group of companies is subject to European data 
protection regulations.



Itiviti’s Privacy Notice is available at 
www.itiviti.com <http://www.itiviti.com/>. Itiviti expects the recipient of 
this email to be compliant with Itiviti’s Privacy Notice and applicable 
regulations. Please advise us immediately at dataprotectionteam@Itiviti.com 
if you are not compliant with these.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message