spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Burak Yavuz <brk...@gmail.com>
Subject Re: netlib-java cannot load native lib in Windows when using spark-submit
Date Sun, 22 Mar 2015 20:15:36 GMT
Did you build Spark with: -Pnetlib-lgpl?

Ref: https://spark.apache.org/docs/latest/mllib-guide.html

Burak

On Sun, Mar 22, 2015 at 7:37 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> How about pointing LD_LIBRARY_PATH to native lib folder ?
>
> You need Spark 1.2.0 or higher for the above to work. See SPARK-1719
>
> Cheers
>
> On Sun, Mar 22, 2015 at 4:02 AM, Xi Shen <davidshen84@gmail.com> wrote:
>
>> Hi Ted,
>>
>> I have tried to invoke the command from both cygwin environment and
>> powershell environment. I still get the messages:
>>
>> 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from:
>> com.github.fommil.netlib.NativeSystemBLAS
>> 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from:
>> com.github.fommil.netlib.NativeRefBLAS
>>
>> From the Spark UI, I can see:
>>
>>   spark.driver.extraLibrary c:\openblas
>>
>>
>> Thanks,
>> David
>>
>>
>> On Sun, Mar 22, 2015 at 11:45 AM Ted Yu <yuzhihong@gmail.com> wrote:
>>
>>> Can you try the --driver-library-path option ?
>>>
>>> spark-submit --driver-library-path /opt/hadoop/lib/native ...
>>>
>>> Cheers
>>>
>>> On Sat, Mar 21, 2015 at 4:58 PM, Xi Shen <davidshen84@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I use the *OpenBLAS* DLL, and have configured my application to work
>>>> in IDE. When I start my Spark application from IntelliJ IDE, I can see in
>>>> the log that the native lib is loaded successfully.
>>>>
>>>> But if I use *spark-submit* to start my application, the native lib
>>>> still cannot be load. I saw the WARN message that it failed to load both
>>>> the native and native-ref library. I checked the *Environment* tab in
>>>> the Spark UI, and the *java.library.path* is set correctly.
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> David
>>>>
>>>>
>>>>
>>>
>

Mime
View raw message