spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdullah Bashir <>
Subject Fwd: [MLlib] RowMatrix computeSVD Native ARPACK support not detecting.
Date Mon, 09 Oct 2017 07:46:44 GMT

I am getting the following Warning when i run the pyspark job:

My Code is

mat = RowMatrix(tf_rdd_vec.cache())  # RDD is cached
svd = mat.computeSVD(num_topics, computeU=False)

I am using Ubuntu 16.04 EC2 instance. And I have installed all following libraries into my

sudo apt install libarpack2 Arpack++ libatlas-base-dev liblapacke-dev libblas-dev gfortran
libblas-dev liblapack-dev libnetlib-java libgfortran3 libatlas3-base libopenblas-base

Now when i list /usr/lib directory it shown me the .so files

ubuntu:~$ ls /usr/lib/*.so | grep "pack\|blas"

I have adjusted LD_LIBRARY_PATH to point to above directory as well.

export LD_LIBRARY_PATH=/var/lib/

But Still I am not able to use the Native ARPACK implementation. Also I am Caching the RDD
passing to matrix But it still throws Cache WARNING Any suggestion how to solve these 3 Warnings

I have downloaded compiled version of spark-2.2.0 from the spark download page.

StackOverflow Link:

Best Regards,

View raw message