kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: Kafka Connect Consumer reading messages from Kafka recursively
Date Tue, 03 Jan 2017 23:21:54 GMT
It's a bit odd (and I just opened a JIRA about it), but you actually need
read permission for the group and read permission for the topic.

There are some error responses which may only be logged at DEBUG level, but
I think they should all be throwing an exception and Kafka Connect would
log that at ERROR level. The only case I can find that doesn't do that is
if topic authorization failed.

-Ewen

On Tue, Jan 3, 2017 at 2:21 PM, Srikrishna Alla <allasrikrishna1@gmail.com>
wrote:

> Hi Ewen,
>
> I did not see any "ERROR" messages in the connect logs. But, I checked the
> __consumer_offsets topic and it doesn't have anything in it. Should I
> provide write permissions to this topic for my Kafka client user? I am
> running my consumer using a different user than Kafka user.
>
> Thanks,
> Sri
>
> On Tue, Jan 3, 2017 at 3:40 PM, Ewen Cheslack-Postava <ewen@confluent.io>
> wrote:
>
> > On Tue, Jan 3, 2017 at 12:58 PM, Srikrishna Alla <
> > allasrikrishna1@gmail.com>
> > wrote:
> >
> > > Thanks for your response Ewen. I will try to make updates to the
> producer
> > > as suggested. Regd the Sink Connector consumer, Could it be that
> > > connect-offsets topic is not getting updated with the offset
> information
> > > per consumer? In that case, will the connector consume the same
> messages
> > > again and again? Also, if that is the case, how would I be able to
> > > troubleshoot? I am running a secured Kafka setup with SASL_PLAINTEXT
> > setup.
> > > Which users/groups should have access to write to the default topics?
> If
> > > not, please guide me in the right direction.
> > >
> > >
> > For sink connectors we actually don't use the connect-offsets topic. So
> if
> > you only have that one sink connector running, you shouldn't expect to
> see
> > any writes to it. Since sink connectors are just consumer groups, they
> use
> > the existing __consumer_offsets topic for storage and do the commits via
> > the normal consumer commit APIs. For ACLs, you'll want Read access to the
> > Group and the Topic.
> >
> > But I doubt it is ACL issues if you're only seeing this when there is
> heavy
> > load. You could use the consumer offset checker to see if any offsets are
> > committed for the group. Also, is there anything in the logs that might
> > indicate a problem with the consumer committing offsets?
> >
> > -Ewen
> >
> >
> > > Thanks,
> > > Sri
> > >
> > > On Tue, Jan 3, 2017 at 1:59 PM, Ewen Cheslack-Postava <
> ewen@confluent.io
> > >
> > > wrote:
> > >
> > > > On Tue, Jan 3, 2017 at 8:38 AM, Srikrishna Alla <
> > > allasrikrishna1@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I am using Kafka/Kafka Connect to track certain events happening
in
> > my
> > > > > application. This is how I have implemented it -
> > > > > 1. My application is opening a KafkaProducer every time this event
> > > > happens
> > > > > and writes to my topic. My application has several components
> running
> > > in
> > > > > Yarn and so I did not find a way to have just one producer and
> reuse
> > > it.
> > > > > Once the event has been published, producer is closed
> > > > >
> > > >
> > > > KafkaProducer is thread safe, so you can allocate a single producer
> per
> > > > process and use it every time the event occurs on any thread.
> Creating
> > > and
> > > > destroying a producer for every event will be very inefficient -- not
> > > only
> > > > are you opening new TCP connections every time, having to lookup
> > metadata
> > > > every time, etc, you also don't allow the producer to get any benefit
> > > from
> > > > batching so every message will require its own request/response.
> > > >
> > > >
> > > > > 2. I am using Kafka Connect Sink Connector to consume from my topic
> > and
> > > > > write to DB and do other processing.
> > > > >
> > > > > This setup is working great as long as we have a stable number of
> > > events
> > > > > published. The issue I am facing is when we have a huge number of
> > > > events(in
> > > > > thousands within minutes) hitting Kafka. In this case, my Sink
> > > Connector
> > > > is
> > > > > going into a loop and reading events from Kafka recursively and not
> > > > > stopping. What could have triggered this? Please provide your
> > valuable
> > > > > insights.
> > > > >
> > > >
> > > > What exactly do you mean by "reading events from Kafka recursively"?
> > > Unless
> > > > it's hitting some errors that are causing consumers to fall out of
> the
> > > > group uncleanly and then rejoin later, you shouldn't be seeing
> > > duplicates.
> > > > Is there anything from the logs that might help reveal the problem?
> > > >
> > > > -Ewen
> > > >
> > > >
> > > > >
> > > > > Thanks,
> > > > > Sri
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message