spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Artemis User <>
Subject Re: Debugging tools for Spark Structured Streaming
Date Fri, 30 Oct 2020 14:59:15 GMT
Spark distribute loads to executors and the executors are usually 
pre-configured with the number of cores.  You may want to check with 
your Spark admin on how many executors (or slaves) your Spark cluster is 
configured with and how many cores are pre-configured for executors.  
The debugging tool for performance tuning in Spark would be the built-in 
Web UI.

The level of parallel processing in structured streaming isn't as 
straightforward as standard ETL processing.  It depends on the data 
source, streaming mode (continuous or microbatch), your trigger timing, 
etc.  We have experienced similar scaling problems with structured 
streaming.  Please note that Spark is designed for processing large data 
chunks, not for streaming type of data one piece at a time.  It doesn't 
like small piece of data (the default partition size is set to 128 MB), 
period!  The partition mechanism and its RDD-driven DAG Job scheduler 
are all designed for processing large-scale data for ETL.  It has to 
accumulate streaming data into a large chunk first, before scaling can 
take place.  Apparently Spark can't distribute the read operation either 
(only one worker, and it has to do with preserving the order of stream 
data).  So your data ingestion becomes a bottleneck that prevents from 
scaling down the chain.   The alternatives may be to look into other 
streaming frameworks, like Apache Ignite..

-- ND

On 10/29/20 8:02 PM, Eric Beabes wrote:
> We're using Spark 2.4. We recently pushed to production a product 
> that's using Spark Structured Streaming. It's working well most of the 
> time but occasionally, when the load is high, we've noticed that there 
> are only 10+ 'Active Tasks' even though we've provided 128 cores. 
> Would like to debug this further. Why are all the Cores not getting 
> used? How do we debug this? Please help. Thanks.

To unsubscribe e-mail:

View raw message