spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Abdullah Bashir <mabdullah...@gmail.com>
Subject Fwd: [MLlib] RowMatrix computeSVD Native ARPACK support not detecting.
Date Mon, 09 Oct 2017 07:46:44 GMT
Hi,

I am getting the following Warning when i run the pyspark job:

My Code is

mat = RowMatrix(tf_rdd_vec.cache())  # RDD is cached
svd = mat.computeSVD(num_topics, computeU=False)

I am using Ubuntu 16.04 EC2 instance. And I have installed all following libraries into my
system.

sudo apt install libarpack2 Arpack++ libatlas-base-dev liblapacke-dev libblas-dev gfortran
libblas-dev liblapack-dev libnetlib-java libgfortran3 libatlas3-base libopenblas-base

Now when i list /usr/lib directory it shown me the .so files

ubuntu:~$ ls /usr/lib/*.so | grep "pack\|blas"
/usr/lib/libarpack.so
/usr/lib/libblas.so
/usr/lib/libcblas.so
/usr/lib/libf77blas.so
/usr/lib/liblapack_atlas.so
/usr/lib/liblapacke.so
/usr/lib/liblapack.so
/usr/lib/libopenblasp-r0.2.18.so
/usr/lib/libopenblas.so
/usr/lib/libparpack.so

I have adjusted LD_LIBRARY_PATH to point to above directory as well.

export LD_LIBRARY_PATH=/var/lib/

But Still I am not able to use the Native ARPACK implementation. Also I am Caching the RDD
passing to matrix But it still throws Cache WARNING Any suggestion how to solve these 3 Warnings
?

I have downloaded compiled version of spark-2.2.0 from the spark download page.

StackOverflow Link: https://stackoverflow.com/questions/46612006/how-to-properly-setup-native-arpack-for-spark-2-2-0

Best Regards,


Mime
View raw message