spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig Ingram <cinple....@gmail.com>
Subject Set spark.*.retained* configs to 0 when the UI is disabled?
Date Fri, 13 Oct 2017 17:36:45 GMT
I was recently debugging an OOM exception one of my coworkers was struggling
with and found that `SQLListener._stageIdToStageMetrics` was the culprit.
The UI was disabled in this case, but stats were still accumulating for
jobs, stages, and tasks. The job my coworker was running had over 40k tasks
in one of the stages. Does it make sense to set different defaults for the
following settings when the UI is disabled?

spark.sql.ui.retainedExecutions
spark.ui.retainedJobs
spark.ui.retainedStages
spark.ui.retainedTasks

There may be some other configuration settings that should change too; but
at a minimum, these settings are all potentially problematic as they can
grow unbounded. Is there a reason these settings are using their default
values even when the UI is disabled? If not, it seems like we could save
users a lot of headaches by setting these values to 0 when the UI is
disabled.

Moreover, how does this work with streaming? It seems like this problem
would come up quite often.

Thanks,
Craig



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message