I couldn't find much literature on this so I figured I could ask here.
Does anyone have experience in tuning the memory settings and interval times of the Spark History Server?
Let's say I have 500 applications at 0.5 G each with a spark.history.fs.update.interval of 400s.
Is there a direct memory correlation that can help me set an optimum value?
Looking for some advice if anyone has tuned the History Server to render large amounts of applications.