spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 刘 勇 <cs...@outlook.com>
Subject Re: spark stream kafka wait for all data process done
Date Fri, 02 Aug 2019 02:37:14 GMT
Hi,
You can set spark.streaming.kafka.backpressure.enable=true.
If your tasks can't process larger data that this variable can control the kafka data into
streaming speed. And you can increment your streaming process time window.



Sent from my Samsung Galaxy smartphone.


-------- Original message --------
From: zenglong chen <czlong.kelvin@gmail.com>
Date: 8/2/19 09:59 (GMT+08:00)
To: user@spark.apache.org
Subject: spark stream kafka wait for all data process done

How can kafka wait for tasks process done then begin receive next batch?I want to process
5000 record once by pandas and it may take too long time to process.
Mime
View raw message