spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Takeshi Yamamuro <linguin....@gmail.com>
Subject Re: freeing up memory occupied by processed Stream Blocks
Date Fri, 20 Jan 2017 07:35:09 GMT
Hi,

AFAIK, the blocks of minibatch RDDs are checked every job finished, and
older blocks automatically removed (See:
https://github.com/apache/spark/blob/master/streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala#L463
).

You can control this behaviour by StreamingContext#remember to some extent.

// maropu


On Fri, Jan 20, 2017 at 3:17 AM, Andrew Milkowski <amgm2006@gmail.com>
wrote:

> hello
>
> using spark 2.0.2  and while running sample streaming app with kinesis
> noticed (in admin ui Storage tab)  "Stream Blocks" for each worker keeps
> climbing up
>
> then also (on same ui page) in Blocks section I see blocks such as below
>
> input-0-1484753367056
>
> that are marked as Memory Serialized
>
> that do not seem to be "released"
>
> above eventually consumes executor memories leading to out of memory
> exception on some
>
> is there a way to "release" these blocks free them up , app is sample m/r
>
> I attempted rdd.unpersist(false) in the code but that did not lead to
> memory free up
>
> thanks much in advance!
>



-- 
---
Takeshi Yamamuro

Mime
View raw message