spark-user mailing list archives

Hi all,

I am having troubles with caching and unpersisting a dataset.
I have a cycle that at each iteration filters my dataset.
I realized that caching every x steps (e.g., 50 steps) gives good performance.

However, after a certain number of caching operations, it seems that the memory used for caching
is filled, so I think I should have to unpersist the old cached dataset.

This is my code:


I tried to use an external variable to cache and unpersist it but it doesn’t seem to solve
the problem (maybe I used it in the wrong way).
Do you kindly have any suggestion?

Thank you for your support!
---
Daniele