spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Sparks <evan.spa...@gmail.com>
Subject Re: MLlib linking error Mac OS X
Date Mon, 20 Oct 2014 14:47:19 GMT
MLlib relies on breeze for much of its linear algebra, which in turn relies on netlib-java.
netlib-java will attempt to load a native BLAS at runtime and then attempt to load it's own
precompiled version. Failing that, it will default back to a Java version that it has built
in. The Java version can be about as fast as a native version for certain operations that
are tricky to optimize (like vector dot products), but MUCH slower for things like matrix/matrix
multiply. Luckily - the code will still work without the native libraries installed, it will
just be slower in some situations. So, you can safely ignore the warnings if all you care
about is correctness. 

The MLlib docs (https://spark.apache.org/docs/latest/mllib-guide.html) provide guidance about
how to link against the native libraries in your application - this will make the warning
messages go away and might speed up your program.

- Evan

> On Oct 20, 2014, at 3:54 AM, npomfret <nick-nabble@snowmonkey.co.uk> wrote:
> 
> I'm getting the same warning on my mac.  Accompanied by what appears to be
> pretty low CPU usage
> (http://apache-spark-user-list.1001560.n3.nabble.com/mlib-model-build-and-low-CPU-usage-td16777.html),
> I wonder if they are connected?
> 
> I've used jblas on a mac several times, it always just works perfectly with
> zero setup.  Maybe the warning is misleading.
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/MLlib-linking-error-Mac-OS-X-tp588p16806.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message