spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiangrui Meng <men...@gmail.com>
Subject Re: MLLIB usage: BLAS dependency warning
Date Wed, 12 Nov 2014 06:47:29 GMT
Could you try "jar tf" on the assembly jar and grep
"netlib-native_system-linux-x86_64.so"? -Xiangrui

On Tue, Nov 11, 2014 at 7:11 PM, jpl <jlefevre@soe.ucsc.edu> wrote:
> Hi,
> I am having trouble using the BLAS libs with the MLLib functions.  I am
> using org.apache.spark.mllib.clustering.KMeans (on a single machine) and
> running the Spark-shell with the kmeans example code (from
> https://spark.apache.org/docs/latest/mllib-clustering.html)  which runs
> successfully but I get the following warning in the log:
>
> WARN netlib.BLAS: Failed to load implementation from:
> com.github.fommil.netlib.NativeSystemBLAS
> WARN netlib.BLAS: Failed to load implementation from:
> com.github.fommil.netlib.NativeRefBLAS
>
> I compiled spark 1.1.0 with mvn -Phadoop-2.4  -Dhadoop.version=2.4.0
> -Pnetlib-lgpl -DskipTests clean package
>
> If anyone could please clarify the steps to get the dependencies correctly
> installed and visible to spark (from
> https://spark.apache.org/docs/latest/mllib-guide.html), that would be
> greatly appreciated.  Using yum, I installed blas.x86_64, lapack.x86_64,
> gcc-gfortran.x86_64, libgfortran.x86_64 and then downloaded Breeze and built
> that successfully with Maven.  I verified that I do have
> /usr/lib/libblas.so.3 and /usr/lib/liblapack.so.3 present on the machine and
> ldconf -p shows these listed.
>
> I also tried adding /usr/lib/ to spark.executor.extraLibraryPath and I
> verified it is present in the Spark webUI environment tab.   I downloaded
> and compiled jblas with mvn clean install, which creates
> jblas-1.2.4-SNAPSHOT.jar, and then also tried adding that to
> spark.executor.extraClassPath but I still get the same WARN message. Maybe
> there are a few simple steps that I am missing?  Thanks a lot.
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLLIB-usage-BLAS-dependency-warning-tp18660.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message