spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Parin Choganwala <>
Subject Spark DIMSUM Memory requirement?
Date Tue, 01 Dec 2015 21:56:15 GMT
Hi All,

I am trying to run RowMatrix.similarity(0.5) on 60K users (n) with 130k features (m) on spark
Using 4 m3.2xlarge 30GB RAM and 8 cores but getting lots of ERROR YarnScheduler: Lost executor
1 on XXX.internal: remote Akka client disassociate

What could be the reason?
Is it shuffle memory that I should increase?

Thank You
Parin Choganwala

View raw message