spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renato Marroquín Mogrovejo <renatoj.marroq...@gmail.com>
Subject Re: Running in cluster mode causes native library linking to fail
Date Wed, 14 Oct 2015 19:54:29 GMT
Sorry Bernardo, I just double checked. I use: System.loadLibrary(....);
Could you also try that?


Renato M.

2015-10-14 21:51 GMT+02:00 Renato Marroquín Mogrovejo <
renatoj.marroquin@gmail.com>:

> Hi Bernardo,
>
> So is this in distributed mode? or single node? Maybe fix the issue with a
> single node first ;)
> You are right that Spark finds the library but not the *.so file. I also
> use System.load(<LIBRARY_NAME>) with LD_LIBRARY_PATH set, and I am able to
> execute without issues. Maybe you'd like to double check paths, env
> variables, or the parameters spark.driver.extraLibraryPath,
> spark.executor.extraLibraryPath.
>
>
> Best,
>
> Renato M.
>
> 2015-10-14 21:40 GMT+02:00 Bernardo Vecchia Stein <
> bernardovstein@gmail.com>:
>
>> Hi Renato,
>>
>> I have done that as well, but so far no luck. I believe spark is finding
>> the library correctly, otherwise the error message would be "no libraryname
>> found" or something like that. The problem seems to be something else, and
>> I'm not sure how to find it.
>>
>> Thanks,
>> Bernardo
>>
>> On 14 October 2015 at 16:28, Renato Marroquín Mogrovejo <
>> renatoj.marroquin@gmail.com> wrote:
>>
>>> You can also try setting the env variable LD_LIBRARY_PATH to point where
>>> your compiled libraries are.
>>>
>>>
>>> Renato M.
>>>
>>> 2015-10-14 21:07 GMT+02:00 Bernardo Vecchia Stein <
>>> bernardovstein@gmail.com>:
>>>
>>>> Hi Deenar,
>>>>
>>>> Yes, the native library is installed on all machines of the cluster. I
>>>> tried a simpler approach by just using System.load() and passing the exact
>>>> path of the library, and things still won't work (I get exactly the same
>>>> error and message).
>>>>
>>>> Any ideas of what might be failing?
>>>>
>>>> Thank you,
>>>> Bernardo
>>>>
>>>> On 14 October 2015 at 02:50, Deenar Toraskar <deenar.toraskar@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Bernardo
>>>>>
>>>>> Is the native library installed on all machines of your cluster and
>>>>> are you setting both the spark.driver.extraLibraryPath and
>>>>> spark.executor.extraLibraryPath ?
>>>>>
>>>>> Deenar
>>>>>
>>>>>
>>>>>
>>>>> On 14 October 2015 at 05:44, Bernardo Vecchia Stein <
>>>>> bernardovstein@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am trying to run some scala code in cluster mode using
>>>>>> spark-submit. This code uses addLibrary to link with a .so that exists
in
>>>>>> the machine, and this library has a function to be called natively
(there's
>>>>>> a native definition as needed in the code).
>>>>>>
>>>>>> The problem I'm facing is: whenever I try to run this code in cluster
>>>>>> mode, spark fails with the following message when trying to execute
the
>>>>>> native function:
>>>>>> java.lang.UnsatisfiedLinkError:
>>>>>> org.name.othername.ClassName.nativeMethod([B[B)[B
>>>>>>
>>>>>> Apparently, the library is being found by spark, but the required
>>>>>> function isn't found.
>>>>>>
>>>>>> When trying to run in client mode, however, this doesn't fail and
>>>>>> everything works as expected.
>>>>>>
>>>>>> Does anybody have any idea of what might be the problem here? Is
>>>>>> there any bug that could be related to this when running in cluster
mode?
>>>>>>
>>>>>> I appreciate any help.
>>>>>> Thanks,
>>>>>> Bernardo
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message