spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vitaliy Pisarev <>
Subject How to address seemingly low core utilization on a spark workload?
Date Thu, 15 Nov 2018 14:51:14 GMT
I have a workload that runs on a cluster of 300 cores.
Below is a plot of the amount of active tasks over time during the
execution of this workload:

[image: image.png]

What I deduce is that there are substantial intervals where the cores are
heavily under-utilised.

What actions can I take to:

   - Increase the efficiency (== core utilisation) of the cluster?
   - Understand the root causes behind the drops in core utilisation?

View raw message