spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Tanase <atan...@adobe.com>
Subject Re: how to broadcast huge lookup table?
Date Sun, 04 Oct 2015 16:34:46 GMT
have a look at .transformWith, you can specify another RDD.

Sent from my iPhone

On 02 Oct 2015, at 21:50, "Saif.A.Ellafi@wellsfargo.com<mailto:Saif.A.Ellafi@wellsfargo.com>"
<Saif.A.Ellafi@wellsfargo.com<mailto:Saif.A.Ellafi@wellsfargo.com>> wrote:

I tried broadcasting a key-value rdd, but then I cannot perform any rdd-actions inside a map/foreach
function of another rdd.

any tips? If going into scala collections I end up with huge memory bottlenecks.

Saif


Mime
View raw message