spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Beabes <>
Subject Understanding Executors UI
Date Tue, 05 Jan 2021 23:19:51 GMT
[image: image.png]

Not sure if this image will go through. (Never sent an email to this
mailing list with an image).

I am trying to understand this 'Executors' UI in Spark 2.4. I have a
Stateful Structured Streaming job with 'State timeout' set to 10 minutes.
When the load on the system is low a message gets written to Kafka
immediately after the State times out BUT under heavy load it takes over 40
minutes to get a message on the output topic. Trying to debug this issue &
see if performance can be improved.


1) I am requesting 3.2 TB of memory but it seems the job keeps using only
598.5 GB as per the values in 'Storage Memory' as well as 'On Heap Storage
Memory'. Wondering if this is a Cluster issue OR am I not setting values
2) Where can I find documentation to understand different 'Tabs' in the
Spark UI? (Sorry, Googling didn't help. I will keep searching.)

Any pointers would be appreciated. Thanks.

View raw message