kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Azama <eazama...@gmail.com>
Subject Re: Consumer hangs at poll() and never throw exception on client
Date Mon, 16 Dec 2019 04:45:39 GMT
Consumers are unable to make any progress if the next records to be sent
are part of an open transaction. Are your producers properly closing
transactions?

On Sun, Dec 15, 2019 at 7:49 PM Frank Zhou <frank.zhou@itiviti.com> wrote:

> More details, the first application's consumer has been closed and its
> producers are running already when the second application starts, then
> second consumer hangs forever if transaction is enabled on producers.
>
> On Mon, Dec 16, 2019 at 11:43 AM Frank Zhou <frank.zhou@itiviti.com>
> wrote:
>
> > Hi Matthias,
> >
> > Seems the issue is related to transaction. Let me expand a bit on our
> flow
> > of this. Our application will use single consumer to fetch the records,
> > deal with the data and then multiple producers to send data afterwards.
> > There could be multiple applications sending data to a single topic as
> > well. So the above issue happens when transactions are enabled when
> > multiple applications are running(Transaction ids are different among
> > producers and applications). Today I tested with transaction disabled,
> and
> > everything seems fine. Do you see any issue for our usage on
> transactions?
> >
> > On Tue, Dec 10, 2019 at 4:42 PM Matthias J. Sax <matthias@confluent.io>
> > wrote:
> >
> >> Hard to say why the second consumer does not fetch data.
> >>
> >> I would try to inspect the logs at DEBUG level to get more insight what
> >> going on.
> >>
> >>
> >> -Matthias
> >>
> >> On 12/10/19 12:27 AM, Frank Zhou wrote:
> >> > "earliest",  as we need everything from the start
> >> >
> >> > On Tue, Dec 10, 2019 at 4:22 PM Matthias J. Sax <
> matthias@confluent.io>
> >> > wrote:
> >> >
> >> >> What is your `auto.offset.reset` policy?
> >> >>
> >> >> On 12/10/19 12:02 AM, Frank Zhou wrote:
> >> >>> Hi,
> >> >>>
> >> >>> Thanks for your prompt reply. Consumers have different group.id
in
> >> our
> >> >>> case, as we just need all the data in the topic, so every time
the
> >> >>> application starts, the consumer will be in a new group. For current
> >> >>> testing environment, we just one partition for one topic. though
we
> >> have
> >> >>> tried with multiple partitions, and situation happened as well.
> >> >>>
> >> >>> On Tue, Dec 10, 2019 at 3:56 PM Matthias J. Sax <
> >> matthias@confluent.io>
> >> >>> wrote:
> >> >>>
> >> >>>> Are both consumers in the same consumer group, ie, use the
same `
> >> >> group.id
> >> >>>> `?
> >> >>>>
> >> >>>> If yes, how many partitions does the topic have? If it has
only one
> >> >>>> partition, the observed behavior is expected, because a single
> >> >>>> partitions can only be read by a single consumer instance per
> >> consumer
> >> >>>> group. The second consumer won't have any assigned partitions
to
> read
> >> >> from.
> >> >>>>
> >> >>>> -Matthias
> >> >>>>
> >> >>>> On 12/9/19 11:12 PM, Frank Zhou wrote:
> >> >>>>> Hi,
> >> >>>>>
> >> >>>>> I am quite new to Kafka, and I have encountered a weird
case
> during
> >> QA
> >> >>>>> stage for our application. Now we have 2 consumers consuming
same
> >> topic
> >> >>>>> in kafka cluster. The first started consumer works fine
and get
> >> closed
> >> >>>>> after getting all the messages. After that, the second
one started
> >> and
> >> >>>>> just hang there in the poll(), which we set the Duration
to 10
> >> seconds,
> >> >>>>> and will quit the loop if no records are fetched. From
kafka
> server
> >> >> log,
> >> >>>>> I don't see warning or error for the second consumer.
> >> >>>>>
> >> >>>>> Attached client/server log related to both consumers. I
am using
> >> 2.3.1
> >> >>>>> client, 2.2.2 kafka servers. Looking forward to you feedback.
> >> >>>>>
> >> >>>>>
> >> >>>>> Best regards,
> >> >>>>> --
> >> >>>>> *Frank Zhou*
> >> >>>>> R&D, Itiviti
> >> >>>>> Java Developer
> >> >>>>> D +852 2521 7480
> >> >>>>> frank.zhou@itiviti.com <mailto:frank.zhou@itiviti.com>
> >> >>>>>
> >> >>>>> ______________________________
> >> >>>>>
> >> >>>>> itiviti.com <https://www.itiviti.com/>
> >> >>>>>
> >> >>>>> /The information contained in or attached to this email
is
> strictly
> >> >>>>> confidential. If you are not the intended recipient, please
notify
> >> us
> >> >>>>> immediately by telephone and return the message to us./
> >> >>>>>
> >> >>>>> /Email communications by definition contain personal information.
> >> The
> >> >>>>> Itiviti group of companies is subject to European data
protection
> >> >>>>> regulations. Itiviti’s Privacy Notice is available at
> >> www.itiviti.com
> >> >>>>> <http://www.itiviti.com/>. Itiviti expects the recipient
of this
> >> email
> >> >>>>> to be compliant with Itiviti’s Privacy Notice and applicable
> >> >>>>> regulations. Please advise us immediately at
> >> >>>>> dataprotectionteam@Itiviti.com if you are not compliant
with
> >> these./
> >> >>>>>
> >> >>>>>
> >> >>>>> ______________________________
> >> >>>>>
> >> >>>>> itiviti.com <https://www.itiviti.com/>
> >> >>>>> <https://www.linkedin.com/company/itiviti> ^Follow
Itiviti on
> >> Linkedin
> >> >>>>> <https://www.linkedin.com/company/10438325/>
> >> >>>>>
> >> >>>>> The information contained in or attached to this email
is strictly
> >> >>>>> confidential. If you are not the intended recipient, please
notify
> >> us
> >> >>>>> immediately by telephone and return the message to us.
Email
> >> >>>>> communications by definition contain personal information.
The
> >> Itiviti
> >> >>>>> group of companies is subject to European data protection
> >> regulations.
> >> >>>>>
> >> >>>>> Itiviti’s Privacy Notice is available at www.itiviti.com
> >> >>>>> <http://www.itiviti.com/>. Itiviti expects the recipient
of this
> >> email
> >> >>>>> to be compliant with Itiviti’s Privacy Notice and applicable
> >> >>>>> regulations. Please advise us immediately at
> >> >>>>> dataprotectionteam@Itiviti.com if you are not compliant
with
> these.
> >> >>>>>
> >> >>>>
> >> >>>>
> >> >>>
> >> >>
> >> >>
> >> >
> >>
> >>
> >
> > --
> > *Frank Zhou*
> > R&D, Itiviti
> > Java Developer
> > D +852 2521 7480
> > frank.zhou@itiviti.com
> >
> > ______________________________
> >
> > itiviti.com <https://www.itiviti.com/>
> >
> > *The information contained in or attached to this email is strictly
> > confidential. If you are not the intended recipient, please notify us
> > immediately by telephone and return the message to us.*
> >
> > *Email communications by definition contain personal information. The
> > Itiviti group of companies is subject to European data protection
> > regulations. Itiviti’s Privacy Notice is available at www.itiviti.com
> > <http://www.itiviti.com/>. Itiviti expects the recipient of this email
> to
> > be compliant with Itiviti’s Privacy Notice and applicable regulations.
> > Please advise us immediately at dataprotectionteam@Itiviti.com if you
> are
> > not compliant with these.*
> >
>
>
> --
> *Frank Zhou*
> R&D, Itiviti
> Java Developer
> D +852 2521 7480
> frank.zhou@itiviti.com
>
> ______________________________
>
> itiviti.com <https://www.itiviti.com/>
>
> *The information contained in or attached to this email is strictly
> confidential. If you are not the intended recipient, please notify us
> immediately by telephone and return the message to us.*
>
> *Email communications by definition contain personal information. The
> Itiviti group of companies is subject to European data protection
> regulations. Itiviti’s Privacy Notice is available at www.itiviti.com
> <http://www.itiviti.com/>. Itiviti expects the recipient of this email to
> be compliant with Itiviti’s Privacy Notice and applicable regulations.
> Please advise us immediately at dataprotectionteam@Itiviti.com if you are
> not compliant with these.*
>
> --
> ______________________________
>
> itiviti.com <https://www.itiviti.com/>
>
> <https://www.linkedin.com/company/itiviti> Follow Itiviti on Linkedin
> <https://www.linkedin.com/company/10438325/>
>
>
>
>
> The information contained
> in or attached to this email is strictly confidential. If you are not the
> intended recipient, please notify us immediately by telephone and return
> the message to us. Email communications by definition contain personal
> information. The Itiviti group of companies is subject to European data
> protection regulations.
>
>
>
> Itiviti’s Privacy Notice is available at
> www.itiviti.com <http://www.itiviti.com/>. Itiviti expects the recipient
> of
> this email to be compliant with Itiviti’s Privacy Notice and applicable
> regulations. Please advise us immediately at dataprotectionteam@Itiviti.com
> if you are not compliant with these.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message