spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Zhemzhitsky <szh.s...@gmail.com>
Subject RDD checkpoint failures in case of insufficient memory
Date Thu, 15 Mar 2018 06:43:25 GMT
Hi there,

A while ago running GraphX jobs I've discovered that
PeriodicRDDCheckpointer fails with FileNotFoundException's in case of
insufficient memory resources.

I believe that any iterative job which uses PeriodicRDDCheckpointer
(like ML) suffers from the same issue (but it's not visible enough
because of RAM size of modern servers).

So, I've raised the JIRA issues with the corresponding pull requests to fix them
- https://issues.apache.org/jira/browse/SPARK-22150
- https://issues.apache.org/jira/browse/SPARK-22184
- https://github.com/apache/spark/pull/19373
- https://github.com/apache/spark/pull/19410

Could anyone please look through these PRs?

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message