Hi, John.
I've had similar problems. IIRC, the driver was GCing madly. I don't know
why the driver was doing so much work but I quickly implemented an
alternative approach. The code I wrote belongs to my client but I wrote
something that should be equivalent. It can be found at:
https://github.com/PhillHenry/Algernon
It's not terribly complicated and you could rollyourown if you prefer
(the rough idea can be found at
http://javaagile.blogspot.co.at/2016/11/analternativeapproachtomatricesin.html).
But anyway, I got good performance this way.
Phill
On Thu, May 11, 2017 at 10:12 PM, John Compitello <johnc@broadinstitute.org>
wrote:
> Hey all,
>
> I’ve found myself in a position where I need to do a relatively large
> matrix multiply (at least, compared to what I normally have to do). I’m
> looking to multiply a 100k by 500k dense matrix by its transpose to yield
> 100k by 100k matrix. I’m trying to do this on Google Cloud, so I don’t have
> any real limits on cluster size or memory. However, I have no idea where to
> begin as far as number of cores / number of partitions / how big to make
> the block size for best performance. Is there anywhere where Spark users
> collect optimal configurations for methods relative to data input size?
> Does anyone have any suggestions? I’ve tried throwing 900 cores at a 100k
> by 100k matrix multiply with 1000 by 1000 sized blocks, and that seemed to
> hang forever and eventually fail.
>
> Thanks ,
>
> John
> 
> To unsubscribe email: userunsubscribe@spark.apache.org
>
>
