kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikrishna Alla <allasrikrish...@gmail.com>
Subject Re: Kafka Connect Consumer reading messages from Kafka recursively
Date Wed, 04 Jan 2017 17:37:19 GMT
Hi Ewen,

My assumption that this issue is happening when a huge number of events are
getting published was wrong. That is when we discovered it. I looked
closely at the log file and I seem to be having this issue for everything.
So, my consumer is reading all the events from Kafka topic and restarting
from the top again. This is happening continuously. I added read/write
permissions for the Kafka client user to __consumer_offsets and restarted
Kafka Connect. I am still facing this issue. Is there anything else I can
check? I did not have this issue when we ran on an unsecured cluster. So, I
keep thinking this has something to do with some permissions issue. Your
help is greatly appreciated.

Thanks,
Sri

On Tue, Jan 3, 2017 at 5:21 PM, Ewen Cheslack-Postava <ewen@confluent.io>
wrote:

> It's a bit odd (and I just opened a JIRA about it), but you actually need
> read permission for the group and read permission for the topic.
>
> There are some error responses which may only be logged at DEBUG level, but
> I think they should all be throwing an exception and Kafka Connect would
> log that at ERROR level. The only case I can find that doesn't do that is
> if topic authorization failed.
>
> -Ewen
>
> On Tue, Jan 3, 2017 at 2:21 PM, Srikrishna Alla <allasrikrishna1@gmail.com
> >
> wrote:
>
> > Hi Ewen,
> >
> > I did not see any "ERROR" messages in the connect logs. But, I checked
> the
> > __consumer_offsets topic and it doesn't have anything in it. Should I
> > provide write permissions to this topic for my Kafka client user? I am
> > running my consumer using a different user than Kafka user.
> >
> > Thanks,
> > Sri
> >
> > On Tue, Jan 3, 2017 at 3:40 PM, Ewen Cheslack-Postava <ewen@confluent.io
> >
> > wrote:
> >
> > > On Tue, Jan 3, 2017 at 12:58 PM, Srikrishna Alla <
> > > allasrikrishna1@gmail.com>
> > > wrote:
> > >
> > > > Thanks for your response Ewen. I will try to make updates to the
> > producer
> > > > as suggested. Regd the Sink Connector consumer, Could it be that
> > > > connect-offsets topic is not getting updated with the offset
> > information
> > > > per consumer? In that case, will the connector consume the same
> > messages
> > > > again and again? Also, if that is the case, how would I be able to
> > > > troubleshoot? I am running a secured Kafka setup with SASL_PLAINTEXT
> > > setup.
> > > > Which users/groups should have access to write to the default topics?
> > If
> > > > not, please guide me in the right direction.
> > > >
> > > >
> > > For sink connectors we actually don't use the connect-offsets topic. So
> > if
> > > you only have that one sink connector running, you shouldn't expect to
> > see
> > > any writes to it. Since sink connectors are just consumer groups, they
> > use
> > > the existing __consumer_offsets topic for storage and do the commits
> via
> > > the normal consumer commit APIs. For ACLs, you'll want Read access to
> the
> > > Group and the Topic.
> > >
> > > But I doubt it is ACL issues if you're only seeing this when there is
> > heavy
> > > load. You could use the consumer offset checker to see if any offsets
> are
> > > committed for the group. Also, is there anything in the logs that might
> > > indicate a problem with the consumer committing offsets?
> > >
> > > -Ewen
> > >
> > >
> > > > Thanks,
> > > > Sri
> > > >
> > > > On Tue, Jan 3, 2017 at 1:59 PM, Ewen Cheslack-Postava <
> > ewen@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > > On Tue, Jan 3, 2017 at 8:38 AM, Srikrishna Alla <
> > > > allasrikrishna1@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I am using Kafka/Kafka Connect to track certain events happening
> in
> > > my
> > > > > > application. This is how I have implemented it -
> > > > > > 1. My application is opening a KafkaProducer every time this
> event
> > > > > happens
> > > > > > and writes to my topic. My application has several components
> > running
> > > > in
> > > > > > Yarn and so I did not find a way to have just one producer and
> > reuse
> > > > it.
> > > > > > Once the event has been published, producer is closed
> > > > > >
> > > > >
> > > > > KafkaProducer is thread safe, so you can allocate a single producer
> > per
> > > > > process and use it every time the event occurs on any thread.
> > Creating
> > > > and
> > > > > destroying a producer for every event will be very inefficient --
> not
> > > > only
> > > > > are you opening new TCP connections every time, having to lookup
> > > metadata
> > > > > every time, etc, you also don't allow the producer to get any
> benefit
> > > > from
> > > > > batching so every message will require its own request/response.
> > > > >
> > > > >
> > > > > > 2. I am using Kafka Connect Sink Connector to consume from my
> topic
> > > and
> > > > > > write to DB and do other processing.
> > > > > >
> > > > > > This setup is working great as long as we have a stable number
of
> > > > events
> > > > > > published. The issue I am facing is when we have a huge number
of
> > > > > events(in
> > > > > > thousands within minutes) hitting Kafka. In this case, my Sink
> > > > Connector
> > > > > is
> > > > > > going into a loop and reading events from Kafka recursively
and
> not
> > > > > > stopping. What could have triggered this? Please provide your
> > > valuable
> > > > > > insights.
> > > > > >
> > > > >
> > > > > What exactly do you mean by "reading events from Kafka
> recursively"?
> > > > Unless
> > > > > it's hitting some errors that are causing consumers to fall out of
> > the
> > > > > group uncleanly and then rejoin later, you shouldn't be seeing
> > > > duplicates.
> > > > > Is there anything from the logs that might help reveal the problem?
> > > > >
> > > > > -Ewen
> > > > >
> > > > >
> > > > > >
> > > > > > Thanks,
> > > > > > Sri
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message