spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From paja <vybi...@gmail.com>
Subject Join stucks in the last stage step
Date Wed, 07 Jan 2015 17:27:54 GMT
Hello,

  I have problem with join of two tables via Spark - I have tried to do it
via Spark SQL and API but no progress so far. I have basicaly two tables
ACCONTS - 16 mio records and TRANSACTIONS 2,5 billion records. When I try to
join the tables (please see code) the job stucks in the last stage for very
long (please see console output).  And after eg 2h it writes to the output a
weird exception like
/org.apache.spark.shuffle.MetadataFetchFailedException: Missing an output
location for shuffle 0/

I have tried several strategies - repartitioning of RDDs, broadcast the
smaller one, but result is always same
Have sombody idea what happens? 

Source Code.  AccJoin.java
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n21018/AccJoin.java>  
Console  AccJoin_0.html
<http://apache-spark-user-list.1001560.n3.nabble.com/file/n21018/AccJoin_0.html>  



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Join-stucks-in-the-last-stage-step-tp21018.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message