spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Parin Choganwala <pa...@7parkdata.com>
Subject Spark DIMSUM Memory requirement?
Date Tue, 01 Dec 2015 21:56:15 GMT
Hi All,

I am trying to run RowMatrix.similarity(0.5) on 60K users (n) with 130k features (m) on spark
1.3.0.
Using 4 m3.2xlarge 30GB RAM and 8 cores but getting lots of ERROR YarnScheduler: Lost executor
1 on XXX.internal: remote Akka client disassociate

What could be the reason?
Is it shuffle memory that I should increase?

Thank You
Parin Choganwala


Mime
View raw message