kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikrishna Alla <allasrikrish...@gmail.com>
Subject Re: Kafka Connect Consumer reading messages from Kafka recursively
Date Tue, 03 Jan 2017 22:21:34 GMT
Hi Ewen,

I did not see any "ERROR" messages in the connect logs. But, I checked the
__consumer_offsets topic and it doesn't have anything in it. Should I
provide write permissions to this topic for my Kafka client user? I am
running my consumer using a different user than Kafka user.

Thanks,
Sri

On Tue, Jan 3, 2017 at 3:40 PM, Ewen Cheslack-Postava <ewen@confluent.io>
wrote:

> On Tue, Jan 3, 2017 at 12:58 PM, Srikrishna Alla <
> allasrikrishna1@gmail.com>
> wrote:
>
> > Thanks for your response Ewen. I will try to make updates to the producer
> > as suggested. Regd the Sink Connector consumer, Could it be that
> > connect-offsets topic is not getting updated with the offset information
> > per consumer? In that case, will the connector consume the same messages
> > again and again? Also, if that is the case, how would I be able to
> > troubleshoot? I am running a secured Kafka setup with SASL_PLAINTEXT
> setup.
> > Which users/groups should have access to write to the default topics? If
> > not, please guide me in the right direction.
> >
> >
> For sink connectors we actually don't use the connect-offsets topic. So if
> you only have that one sink connector running, you shouldn't expect to see
> any writes to it. Since sink connectors are just consumer groups, they use
> the existing __consumer_offsets topic for storage and do the commits via
> the normal consumer commit APIs. For ACLs, you'll want Read access to the
> Group and the Topic.
>
> But I doubt it is ACL issues if you're only seeing this when there is heavy
> load. You could use the consumer offset checker to see if any offsets are
> committed for the group. Also, is there anything in the logs that might
> indicate a problem with the consumer committing offsets?
>
> -Ewen
>
>
> > Thanks,
> > Sri
> >
> > On Tue, Jan 3, 2017 at 1:59 PM, Ewen Cheslack-Postava <ewen@confluent.io
> >
> > wrote:
> >
> > > On Tue, Jan 3, 2017 at 8:38 AM, Srikrishna Alla <
> > allasrikrishna1@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I am using Kafka/Kafka Connect to track certain events happening in
> my
> > > > application. This is how I have implemented it -
> > > > 1. My application is opening a KafkaProducer every time this event
> > > happens
> > > > and writes to my topic. My application has several components running
> > in
> > > > Yarn and so I did not find a way to have just one producer and reuse
> > it.
> > > > Once the event has been published, producer is closed
> > > >
> > >
> > > KafkaProducer is thread safe, so you can allocate a single producer per
> > > process and use it every time the event occurs on any thread. Creating
> > and
> > > destroying a producer for every event will be very inefficient -- not
> > only
> > > are you opening new TCP connections every time, having to lookup
> metadata
> > > every time, etc, you also don't allow the producer to get any benefit
> > from
> > > batching so every message will require its own request/response.
> > >
> > >
> > > > 2. I am using Kafka Connect Sink Connector to consume from my topic
> and
> > > > write to DB and do other processing.
> > > >
> > > > This setup is working great as long as we have a stable number of
> > events
> > > > published. The issue I am facing is when we have a huge number of
> > > events(in
> > > > thousands within minutes) hitting Kafka. In this case, my Sink
> > Connector
> > > is
> > > > going into a loop and reading events from Kafka recursively and not
> > > > stopping. What could have triggered this? Please provide your
> valuable
> > > > insights.
> > > >
> > >
> > > What exactly do you mean by "reading events from Kafka recursively"?
> > Unless
> > > it's hitting some errors that are causing consumers to fall out of the
> > > group uncleanly and then rejoin later, you shouldn't be seeing
> > duplicates.
> > > Is there anything from the logs that might help reveal the problem?
> > >
> > > -Ewen
> > >
> > >
> > > >
> > > > Thanks,
> > > > Sri
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message