kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: Kafka Connect Consumer reading messages from Kafka recursively
Date Tue, 03 Jan 2017 21:40:49 GMT
On Tue, Jan 3, 2017 at 12:58 PM, Srikrishna Alla <allasrikrishna1@gmail.com>
wrote:

> Thanks for your response Ewen. I will try to make updates to the producer
> as suggested. Regd the Sink Connector consumer, Could it be that
> connect-offsets topic is not getting updated with the offset information
> per consumer? In that case, will the connector consume the same messages
> again and again? Also, if that is the case, how would I be able to
> troubleshoot? I am running a secured Kafka setup with SASL_PLAINTEXT setup.
> Which users/groups should have access to write to the default topics? If
> not, please guide me in the right direction.
>
>
For sink connectors we actually don't use the connect-offsets topic. So if
you only have that one sink connector running, you shouldn't expect to see
any writes to it. Since sink connectors are just consumer groups, they use
the existing __consumer_offsets topic for storage and do the commits via
the normal consumer commit APIs. For ACLs, you'll want Read access to the
Group and the Topic.

But I doubt it is ACL issues if you're only seeing this when there is heavy
load. You could use the consumer offset checker to see if any offsets are
committed for the group. Also, is there anything in the logs that might
indicate a problem with the consumer committing offsets?

-Ewen


> Thanks,
> Sri
>
> On Tue, Jan 3, 2017 at 1:59 PM, Ewen Cheslack-Postava <ewen@confluent.io>
> wrote:
>
> > On Tue, Jan 3, 2017 at 8:38 AM, Srikrishna Alla <
> allasrikrishna1@gmail.com
> > >
> > wrote:
> >
> > > Hi,
> > >
> > > I am using Kafka/Kafka Connect to track certain events happening in my
> > > application. This is how I have implemented it -
> > > 1. My application is opening a KafkaProducer every time this event
> > happens
> > > and writes to my topic. My application has several components running
> in
> > > Yarn and so I did not find a way to have just one producer and reuse
> it.
> > > Once the event has been published, producer is closed
> > >
> >
> > KafkaProducer is thread safe, so you can allocate a single producer per
> > process and use it every time the event occurs on any thread. Creating
> and
> > destroying a producer for every event will be very inefficient -- not
> only
> > are you opening new TCP connections every time, having to lookup metadata
> > every time, etc, you also don't allow the producer to get any benefit
> from
> > batching so every message will require its own request/response.
> >
> >
> > > 2. I am using Kafka Connect Sink Connector to consume from my topic and
> > > write to DB and do other processing.
> > >
> > > This setup is working great as long as we have a stable number of
> events
> > > published. The issue I am facing is when we have a huge number of
> > events(in
> > > thousands within minutes) hitting Kafka. In this case, my Sink
> Connector
> > is
> > > going into a loop and reading events from Kafka recursively and not
> > > stopping. What could have triggered this? Please provide your valuable
> > > insights.
> > >
> >
> > What exactly do you mean by "reading events from Kafka recursively"?
> Unless
> > it's hitting some errors that are causing consumers to fall out of the
> > group uncleanly and then rejoin later, you shouldn't be seeing
> duplicates.
> > Is there anything from the logs that might help reveal the problem?
> >
> > -Ewen
> >
> >
> > >
> > > Thanks,
> > > Sri
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message