Hi everyone,

I wonder when DataFrame's unpersist actually happens?
I am using dynamic allocation and I set properties as follows

spark.dynamicAllocation.cachedExecutorIdleTimeout=30s
spark.dynamicAllocation.executorIdleTimeout=30s

If I run following code

var df = sqlContext.read.parquet("s3://test/audience-log")
df.persist
df.count
df.unpersist

The allocated executor is not removed for a long time or even I am not sure if it happens.

but I add df.count at the end of the code, the executor is removed as soon as 30s passes after completing the code.

var df = sqlContext.read.parquet("s3://test/audience-log")
df.persist
df.count
df.unpersist
df.count

I can see following log

 INFO - Requesting 1 new executor because tasks are backlogged (new desired total will be 1)
 INFO - Requesting to kill executor(s) 1
 INFO - Removing executor 1 because it has been idle for 30 seconds (new desired total will be 0)

Regards,
Soonoh