spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashic Mahtab <>
Subject Spark join and large temp files
Date Mon, 08 Aug 2016 18:17:29 GMT
Hello,We have two parquet inputs of the following form:
a: id:String, Name:String  (1.5TB)b: id:String, Number:Int  (1.3GB)
We need to join these two to get (id, Number, Name). We've tried two approaches:
a.join(b, Seq("id"), "right_outer")
where a and b are dataframes. We also tried taking the rdds, mapping them to pair rdds with
id as the key, and then joining. What we're seeing is that temp file usage is increasing on
the join stage, and filling up our disks, causing the job to crash. Is there a way to join
these two data sets without well...crashing?
Note, the ids are unique, and there's a one to one mapping between the two datasets. 
Any help would be appreciated.

View raw message