spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiangrui Meng <men...@gmail.com>
Subject Re: MLLIB and Openblas library in non-default dir
Date Wed, 07 Jan 2015 05:13:15 GMT
spark-submit may not share the same JVM with Spark master and executors.

On Tue, Jan 6, 2015 at 11:40 AM, Tomas Hudik <xhudik@gmail.com> wrote:
> thanks Xiangrui
>
> I'll try it.
>
> BTW: spark-submit is a standalone program (bin/spark-submit). Therefore, JVM
> has to be executed after spark-submit script
> Am I correct?
>
>
>
> On Mon, Jan 5, 2015 at 10:35 PM, Xiangrui Meng <mengxr@gmail.com> wrote:
>>
>> It might be hard to do that with spark-submit, because the executor
>> JVMs may be already up and running before a user runs spark-submit.
>> You can try to use `System.setProperty` to change the property at
>> runtime, though it doesn't seem to be a good solution. -Xiangrui
>>
>> On Fri, Jan 2, 2015 at 6:28 AM, xhudik <xhudik@gmail.com> wrote:
>> > Hi
>> > I have compiled OpenBlas library into nonstandard directory and I want
>> > to
>> > inform Spark app about it via:
>> >
>> > -Dcom.github.fommil.netlib.NativeSystemBLAS.natives=/usr/local/lib/libopenblas.so
>> > which is a standard option in netlib-java
>> > (https://github.com/fommil/netlib-java)
>> >
>> > I tried 2 ways:
>> > 1. via *--conf* parameter
>> > /bin/spark-submit -v  --class
>> > org.apache.spark.examples.mllib.LinearRegression *--conf
>> >
>> > "-Dcom.github.fommil.netlib.NativeSystemBLAS.natives=/usr/local/lib/libopenblas.so"*
>> > examples/target/scala-2.10/spark-examples-1.3.0-SNAPSHOT-hadoop1.0.4.jar
>> > data/mllib/sample_libsvm_data.txt/
>> >
>> > 2. via *--driver-java-options* parameter
>> > /bin/spark-submit -v *--driver-java-options
>> >
>> > "-Dcom.github.fommil.netlib.NativeSystemBLAS.natives=/usr/local/lib/libopenblas.so"*
>> > --class org.apache.spark.examples.mllib.LinearRegression
>> > examples/target/scala-2.10/spark-examples-1.3.0-SNAPSHOT-hadoop1.0.4.jar
>> > data/mllib/sample_libsvm_data.txt
>> > /
>> >
>> > How can I force spark-submit to propagate info about non-standard
>> > placement
>> > of openblas library to netlib-java lib?
>> >
>> > thanks, Tomas
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-spark-user-list.1001560.n3.nabble.com/MLLIB-and-Openblas-library-in-non-default-dir-tp20943.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message