spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom Vacek <minnesota...@gmail.com>
Subject Re: Stage failures
Date Thu, 24 Oct 2013 17:59:51 GMT
I've figured out what the problem is, but I don't understand why.  I'm
hoping somebody can explain this:

(in the spark shell)
val lb = sc.broadcast( (1 to 10000000).toSet)
val breakMe = sc.parallelize(1 to 250).mapPartitions( it => {val
serializedSet = lb.value.toString; Array(0).iterator}).count  //works great

val ll = (1 to 10000000).toSet
val lb = sc.broadcast(ll)
val breakMe = sc.parallelize(1 to 250).mapPartitions( it => {val
serializedSet = lb.value.toString; Array(0).iterator}).count  //Crashes
ignominiously

Mime
View raw message