spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jung, Soonoh" <soonoh.j...@gmail.com>
Subject Fwd: Dynamic allocation - when DataFrame's unpersist actually happens?
Date Thu, 06 Oct 2016 03:45:30 GMT
Hi everyone,

I wonder when DataFrame's unpersist actually happens?
I am using dynamic allocation and I set properties as follows

spark.dynamicAllocation.cachedExecutorIdleTimeout=30s
spark.dynamicAllocation.executorIdleTimeout=30s

If I run following code

var df = sqlContext.read.parquet("s3://test/audience-log")
df.persist
df.count
df.unpersist

The allocated executor is not removed for a long time or even I am not sure
if it happens.

but I add df.count at the end of the code, the executor is removed as soon
as 30s passes after completing the code.

var df = sqlContext.read.parquet("s3://test/audience-log")
df.persist
df.count
df.unpersist
df.count

I can see following log

 INFO - Requesting 1 new executor because tasks are backlogged (new desired
total will be 1)
 INFO - Requesting to kill executor(s) 1
 INFO - Removing executor 1 because it has been idle for 30 seconds (new
desired total will be 0)

Regards,
Soonoh

Mime
View raw message