spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guillaume Pitel <>
Subject Re: Mllib native netlib-java/OpenBLAS
Date Wed, 10 Dec 2014 10:18:33 GMT

I had the same problem, and tried to compile with mvn -Pnetlib-lgpl

$ mvn -Pnetlib-lgpl -Pyarn -Phadoop-2.3 -Dhadoop.version=2.3.0 -DskipTests clean 

Unfortunately, the resulting assembly jar still lacked the netlib-system class. 
This command :

$ jar tvf assembly/target/scala-2.10/spark-assembly-1.1.1-hadoop2.3.0.jar |grep 
netlib | grep Native

returns nothing...

(and for some reason, including the netlib-all in my shipped jar did not solve 
the problem either, apparently the classloader does not find the class)

In Spark, the profile is defined in mllib submodule, but the -Pnetlib-lgpl seems 
not to be transmitted to the child from the parent pom.xml

I don't know how to fix that cleanly (I just added 
<activeByDefault>true</activeByDefault> in mllib's pom.xml), maybe it's just a

problem with my maven version (3.0.5)


> I tried building Spark from the source, by downloading it and running:
> mvn -Pnetlib-lgpl -DskipTests clean package


*Guillaume PITEL, Président*
+33(0)626 222 431

eXenSa S.A.S. <>
41, rue Périer - 92120 Montrouge - FRANCE
Tel +33(0)184 163 677 / Fax +33(0)972 283 705

View raw message