spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roman Pastukhov <>
Subject Removing broadcasts
Date Tue, 03 Dec 2013 14:00:17 GMT

In iterative processes that use broadcasts they seem to cause memory usage
problems as they are left it memory. Unfortunately only way to remove them
now requires reflection hacks.

TTL based cleaning would also remove JobConf broadcasts, moreover it
requires each iteration to perform within some predefined time frame, so it
does not seem like a good option.

So I was wondering what happened to whether it makes sense to
submit similar pull requests?

PS.TTL cleanup also removes broadcast files on disk, does this mean that if
some RDD part that used some old broadcast needs to be recalculated because
of lost executor this will fail?

View raw message