spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Ott <>
Subject Re: Spark structured streaming - performance tuning
Date Fri, 17 Apr 2020 18:06:15 GMT has quite good information
on it.  For Kafka, you need to start with approximation that processing of
each partition is a separate task that need to be executed, so you need to
plan number of cores correspondingly.

Srinivas V  at "Thu, 16 Apr 2020 22:49:15 +0530" wrote:
 SV> Hello, 
 SV> Can someone point me to a good video or document which takes about performance tuning
for structured streaming app? 
 SV> I am looking especially for listening to Kafka topics say 5 topics each with 100 portions
 SV> Trying to figure out best cluster size and number of executors and cores required. 

With best wishes,                    Alex Ott
Twitter: alexott_en (English), alexott (Russian)

To unsubscribe e-mail:

View raw message