kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "LEE, Tung-Yu" <tun...@gmail.com>
Subject Re: Sometimes KafkaProducer.send() is blocked without throwing any Exception
Date Wed, 05 Sep 2018 02:46:13 GMT
Actually we are not sure how often the blocking happens. Whenever it
happens, it seems never to stop until we kill the process.



We start 12 processes  at the same time using KafkaProducer to send data
which is a csv containing around 80,000 records every minute. (crontab)



Like this:

*/1 * * * *  java -jar ./kafka-producer.jar ./producer.properties
./folder01/test.csv

*/1 * * * *  java -jar ./kafka-producer.jar ./producer.properties
./folder02/test.csv

*/1 * * * *  java -jar ./kafka-producer.jar ./producer.properties
./folder03/test.csv

…

*/1 * * * *  java -jar ./kafka-producer.jar ./producer.properties
./folder12/test.csv



We deliberately use same data to test it. In normal case, every process
will be executed and done. However we’ve noticed that sometimes some
process just blocked and never stop. It happens not very often, maybe from
few hours to one or few days and seems that it has no fixed frequency.



We have considered using “Flushing” immediately after sending data but
concerned about increasing latency therefore we didn’t try it. Will give it
a try.


Any thoughts? Thanks!

M. Manna <manmedia@gmail.com> 於 2018年9月4日 週二 18:17 寫道:

> When you say "Blocked" - for how long did the blocking happened? Also, have
> you considered "Flushing" for immediate availability?
>
>
>
> On Tue, 4 Sep 2018 at 11:03, LEE, Tung-Yu <tungyu@gmail.com> wrote:
>
> > Hello,
> >
> > We currently use Kafka 1.0.2 and find that sometimes when using
> > KafkaProducer.send(), it blocked and didn't throw any exception.
> >
> >
> >
> > Some code snippets and configuration are as fellows.
> >
> > Any feedback is welcomed, thank you.
> >
> >
> >
> > Tung-Yu
> >
> >
> >
> > ##### code snippets ########
> >
> >
> >
> > Producer<String, String> producer = new KafkaProducer<>(props);
> >
> >                         try (BufferedReader br = new BufferedReader(new
> > FileReader(new File(targetFilePath)))) {
> >
> >                                     String line = br.readLine();
> >
> >                                     while (line != null) {
> >
> >                                                 ProducerRecord<String,
> > String> data = new ProducerRecord<String, String>(topicName, line);
> >
> >                                                 producer.send(data); //
> > sometimes it blocked here
> >
> >                                                 line = br.readLine();
> >
> >                                     }
> >
> >                         } catch (Exception e) {
> >
> >                                     logger.error(e.getMessage(), e);
> >
> >                         }
> >
> > producer.close();
> >
> >
> >
> > ##### code snippet ########
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > #### some producer configuration ######
> >
> >
> >
> > request.timeout.ms=60000
> >
> > batch.size=20000
> >
> > buffer.memory=33554432
> >
> > retries=3
> >
> >
> >
> > #### some producer configuration ######
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message