spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: netlib-java cannot load native lib in Windows when using spark-submit
Date Sun, 22 Mar 2015 14:37:58 GMT
How about pointing LD_LIBRARY_PATH to native lib folder ?

You need Spark 1.2.0 or higher for the above to work. See SPARK-1719

Cheers

On Sun, Mar 22, 2015 at 4:02 AM, Xi Shen <davidshen84@gmail.com> wrote:

> Hi Ted,
>
> I have tried to invoke the command from both cygwin environment and
> powershell environment. I still get the messages:
>
> 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from:
> com.github.fommil.netlib.NativeSystemBLAS
> 15/03/22 21:56:00 WARN netlib.BLAS: Failed to load implementation from:
> com.github.fommil.netlib.NativeRefBLAS
>
> From the Spark UI, I can see:
>
>   spark.driver.extraLibrary c:\openblas
>
>
> Thanks,
> David
>
>
> On Sun, Mar 22, 2015 at 11:45 AM Ted Yu <yuzhihong@gmail.com> wrote:
>
>> Can you try the --driver-library-path option ?
>>
>> spark-submit --driver-library-path /opt/hadoop/lib/native ...
>>
>> Cheers
>>
>> On Sat, Mar 21, 2015 at 4:58 PM, Xi Shen <davidshen84@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I use the *OpenBLAS* DLL, and have configured my application to work in
>>> IDE. When I start my Spark application from IntelliJ IDE, I can see in the
>>> log that the native lib is loaded successfully.
>>>
>>> But if I use *spark-submit* to start my application, the native lib
>>> still cannot be load. I saw the WARN message that it failed to load both
>>> the native and native-ref library. I checked the *Environment* tab in
>>> the Spark UI, and the *java.library.path* is set correctly.
>>>
>>>
>>> Thanks,
>>>
>>> David
>>>
>>>
>>>
>>

Mime
View raw message