kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Srikrishna Alla <allasrikrish...@gmail.com>
Subject Kafka Connect Consumer reading messages from Kafka recursively
Date Tue, 03 Jan 2017 16:38:28 GMT
Hi,

I am using Kafka/Kafka Connect to track certain events happening in my
application. This is how I have implemented it -
1. My application is opening a KafkaProducer every time this event happens
and writes to my topic. My application has several components running in
Yarn and so I did not find a way to have just one producer and reuse it.
Once the event has been published, producer is closed
2. I am using Kafka Connect Sink Connector to consume from my topic and
write to DB and do other processing.

This setup is working great as long as we have a stable number of events
published. The issue I am facing is when we have a huge number of events(in
thousands within minutes) hitting Kafka. In this case, my Sink Connector is
going into a loop and reading events from Kafka recursively and not
stopping. What could have triggered this? Please provide your valuable
insights.

Thanks,
Sri

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message