kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Zhou <frank.z...@itiviti.com>
Subject Re: Consumer hangs at poll() and never throw exception on client
Date Mon, 16 Dec 2019 05:45:00 GMT
Hi,

You mean commitTransaction() or abortTransaction()? If so, then yes, either
1 will be called for each transaction.

On Mon, Dec 16, 2019 at 12:46 PM Eric Azama <eazama001@gmail.com> wrote:

> Consumers are unable to make any progress if the next records to be sent
> are part of an open transaction. Are your producers properly closing
> transactions?
>
> On Sun, Dec 15, 2019 at 7:49 PM Frank Zhou <frank.zhou@itiviti.com> wrote:
>
> > More details, the first application's consumer has been closed and its
> > producers are running already when the second application starts, then
> > second consumer hangs forever if transaction is enabled on producers.
> >
> > On Mon, Dec 16, 2019 at 11:43 AM Frank Zhou <frank.zhou@itiviti.com>
> > wrote:
> >
> > > Hi Matthias,
> > >
> > > Seems the issue is related to transaction. Let me expand a bit on our
> > flow
> > > of this. Our application will use single consumer to fetch the records,
> > > deal with the data and then multiple producers to send data afterwards.
> > > There could be multiple applications sending data to a single topic as
> > > well. So the above issue happens when transactions are enabled when
> > > multiple applications are running(Transaction ids are different among
> > > producers and applications). Today I tested with transaction disabled,
> > and
> > > everything seems fine. Do you see any issue for our usage on
> > transactions?
> > >
> > > On Tue, Dec 10, 2019 at 4:42 PM Matthias J. Sax <matthias@confluent.io
> >
> > > wrote:
> > >
> > >> Hard to say why the second consumer does not fetch data.
> > >>
> > >> I would try to inspect the logs at DEBUG level to get more insight
> what
> > >> going on.
> > >>
> > >>
> > >> -Matthias
> > >>
> > >> On 12/10/19 12:27 AM, Frank Zhou wrote:
> > >> > "earliest",  as we need everything from the start
> > >> >
> > >> > On Tue, Dec 10, 2019 at 4:22 PM Matthias J. Sax <
> > matthias@confluent.io>
> > >> > wrote:
> > >> >
> > >> >> What is your `auto.offset.reset` policy?
> > >> >>
> > >> >> On 12/10/19 12:02 AM, Frank Zhou wrote:
> > >> >>> Hi,
> > >> >>>
> > >> >>> Thanks for your prompt reply. Consumers have different group.id
> in
> > >> our
> > >> >>> case, as we just need all the data in the topic, so every
time the
> > >> >>> application starts, the consumer will be in a new group. For
> current
> > >> >>> testing environment, we just one partition for one topic.
though
> we
> > >> have
> > >> >>> tried with multiple partitions, and situation happened as
well.
> > >> >>>
> > >> >>> On Tue, Dec 10, 2019 at 3:56 PM Matthias J. Sax <
> > >> matthias@confluent.io>
> > >> >>> wrote:
> > >> >>>
> > >> >>>> Are both consumers in the same consumer group, ie, use
the same `
> > >> >> group.id
> > >> >>>> `?
> > >> >>>>
> > >> >>>> If yes, how many partitions does the topic have? If it
has only
> one
> > >> >>>> partition, the observed behavior is expected, because
a single
> > >> >>>> partitions can only be read by a single consumer instance
per
> > >> consumer
> > >> >>>> group. The second consumer won't have any assigned partitions
to
> > read
> > >> >> from.
> > >> >>>>
> > >> >>>> -Matthias
> > >> >>>>
> > >> >>>> On 12/9/19 11:12 PM, Frank Zhou wrote:
> > >> >>>>> Hi,
> > >> >>>>>
> > >> >>>>> I am quite new to Kafka, and I have encountered a
weird case
> > during
> > >> QA
> > >> >>>>> stage for our application. Now we have 2 consumers
consuming
> same
> > >> topic
> > >> >>>>> in kafka cluster. The first started consumer works
fine and get
> > >> closed
> > >> >>>>> after getting all the messages. After that, the second
one
> started
> > >> and
> > >> >>>>> just hang there in the poll(), which we set the Duration
to 10
> > >> seconds,
> > >> >>>>> and will quit the loop if no records are fetched.
From kafka
> > server
> > >> >> log,
> > >> >>>>> I don't see warning or error for the second consumer.
> > >> >>>>>
> > >> >>>>> Attached client/server log related to both consumers.
I am using
> > >> 2.3.1
> > >> >>>>> client, 2.2.2 kafka servers. Looking forward to you
feedback.
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> Best regards,
> > >> >>>>> --
> > >> >>>>> *Frank Zhou*
> > >> >>>>> R&D, Itiviti
> > >> >>>>> Java Developer
> > >> >>>>> D +852 2521 7480
> > >> >>>>> frank.zhou@itiviti.com <mailto:frank.zhou@itiviti.com>
> > >> >>>>>
> > >> >>>>> ______________________________
> > >> >>>>>
> > >> >>>>> itiviti.com <https://www.itiviti.com/>
> > >> >>>>>
> > >> >>>>> /The information contained in or attached to this
email is
> > strictly
> > >> >>>>> confidential. If you are not the intended recipient,
please
> notify
> > >> us
> > >> >>>>> immediately by telephone and return the message to
us./
> > >> >>>>>
> > >> >>>>> /Email communications by definition contain personal
> information.
> > >> The
> > >> >>>>> Itiviti group of companies is subject to European
data
> protection
> > >> >>>>> regulations. Itiviti’s Privacy Notice is available
at
> > >> www.itiviti.com
> > >> >>>>> <http://www.itiviti.com/>. Itiviti expects the
recipient of
> this
> > >> email
> > >> >>>>> to be compliant with Itiviti’s Privacy Notice and
applicable
> > >> >>>>> regulations. Please advise us immediately at
> > >> >>>>> dataprotectionteam@Itiviti.com if you are not compliant
with
> > >> these./
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> ______________________________
> > >> >>>>>
> > >> >>>>> itiviti.com <https://www.itiviti.com/>
> > >> >>>>> <https://www.linkedin.com/company/itiviti> ^Follow
Itiviti on
> > >> Linkedin
> > >> >>>>> <https://www.linkedin.com/company/10438325/>
> > >> >>>>>
> > >> >>>>> The information contained in or attached to this email
is
> strictly
> > >> >>>>> confidential. If you are not the intended recipient,
please
> notify
> > >> us
> > >> >>>>> immediately by telephone and return the message to
us. Email
> > >> >>>>> communications by definition contain personal information.
The
> > >> Itiviti
> > >> >>>>> group of companies is subject to European data protection
> > >> regulations.
> > >> >>>>>
> > >> >>>>> Itiviti’s Privacy Notice is available at www.itiviti.com
> > >> >>>>> <http://www.itiviti.com/>. Itiviti expects the
recipient of
> this
> > >> email
> > >> >>>>> to be compliant with Itiviti’s Privacy Notice and
applicable
> > >> >>>>> regulations. Please advise us immediately at
> > >> >>>>> dataprotectionteam@Itiviti.com if you are not compliant
with
> > these.
> > >> >>>>>
> > >> >>>>
> > >> >>>>
> > >> >>>
> > >> >>
> > >> >>
> > >> >
> > >>
> > >>
> > >
> > > --
> > > *Frank Zhou*
> > > R&D, Itiviti
> > > Java Developer
> > > D +852 2521 7480
> > > frank.zhou@itiviti.com
> > >
> > > ______________________________
> > >
> > > itiviti.com <https://www.itiviti.com/>
> > >
> > > *The information contained in or attached to this email is strictly
> > > confidential. If you are not the intended recipient, please notify us
> > > immediately by telephone and return the message to us.*
> > >
> > > *Email communications by definition contain personal information. The
> > > Itiviti group of companies is subject to European data protection
> > > regulations. Itiviti’s Privacy Notice is available at www.itiviti.com
> > > <http://www.itiviti.com/>. Itiviti expects the recipient of this email
> > to
> > > be compliant with Itiviti’s Privacy Notice and applicable regulations.
> > > Please advise us immediately at dataprotectionteam@Itiviti.com if you
> > are
> > > not compliant with these.*
> > >
> >
> >
> > --
> > *Frank Zhou*
> > R&D, Itiviti
> > Java Developer
> > D +852 2521 7480
> > frank.zhou@itiviti.com
> >
> > ______________________________
> >
> > itiviti.com <https://www.itiviti.com/>
> >
> > *The information contained in or attached to this email is strictly
> > confidential. If you are not the intended recipient, please notify us
> > immediately by telephone and return the message to us.*
> >
> > *Email communications by definition contain personal information. The
> > Itiviti group of companies is subject to European data protection
> > regulations. Itiviti’s Privacy Notice is available at www.itiviti.com
> > <http://www.itiviti.com/>. Itiviti expects the recipient of this email
> to
> > be compliant with Itiviti’s Privacy Notice and applicable regulations.
> > Please advise us immediately at dataprotectionteam@Itiviti.com if you
> are
> > not compliant with these.*
> >
> > --
> > ______________________________
> >
> > itiviti.com <https://www.itiviti.com/>
> >
> > <https://www.linkedin.com/company/itiviti> Follow Itiviti on Linkedin
> > <https://www.linkedin.com/company/10438325/>
> >
> >
> >
> >
> > The information contained
> > in or attached to this email is strictly confidential. If you are not the
> > intended recipient, please notify us immediately by telephone and return
> > the message to us. Email communications by definition contain personal
> > information. The Itiviti group of companies is subject to European data
> > protection regulations.
> >
> >
> >
> > Itiviti’s Privacy Notice is available at
> > www.itiviti.com <http://www.itiviti.com/>. Itiviti expects the recipient
> > of
> > this email to be compliant with Itiviti’s Privacy Notice and applicable
> > regulations. Please advise us immediately at
> dataprotectionteam@Itiviti.com
> > if you are not compliant with these.
> >
>


-- 
*Frank Zhou*
R&D, Itiviti
Java Developer
D +852 2521 7480
frank.zhou@itiviti.com

______________________________

itiviti.com <https://www.itiviti.com/>

*The information contained in or attached to this email is strictly
confidential. If you are not the intended recipient, please notify us
immediately by telephone and return the message to us.*

*Email communications by definition contain personal information. The
Itiviti group of companies is subject to European data protection
regulations. Itiviti’s Privacy Notice is available at www.itiviti.com
<http://www.itiviti.com/>. Itiviti expects the recipient of this email to
be compliant with Itiviti’s Privacy Notice and applicable regulations.
Please advise us immediately at dataprotectionteam@Itiviti.com if you are
not compliant with these.*

-- 
______________________________

itiviti.com <https://www.itiviti.com/>
 
<https://www.linkedin.com/company/itiviti> Follow Itiviti on Linkedin 
<https://www.linkedin.com/company/10438325/>




The information contained 
in or attached to this email is strictly confidential. If you are not the 
intended recipient, please notify us immediately by telephone and return 
the message to us. Email communications by definition contain personal 
information. The Itiviti group of companies is subject to European data 
protection regulations.



Itiviti’s Privacy Notice is available at 
www.itiviti.com <http://www.itiviti.com/>. Itiviti expects the recipient of 
this email to be compliant with Itiviti’s Privacy Notice and applicable 
regulations. Please advise us immediately at dataprotectionteam@Itiviti.com 
if you are not compliant with these.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message