spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Renato Marroquín Mogrovejo <renatoj.marroq...@gmail.com>
Subject Re: Running in cluster mode causes native library linking to fail
Date Wed, 14 Oct 2015 19:51:39 GMT
Hi Bernardo,

So is this in distributed mode? or single node? Maybe fix the issue with a
single node first ;)
You are right that Spark finds the library but not the *.so file. I also
use System.load(<LIBRARY_NAME>) with LD_LIBRARY_PATH set, and I am able to
execute without issues. Maybe you'd like to double check paths, env
variables, or the parameters spark.driver.extraLibraryPath,
spark.executor.extraLibraryPath.


Best,

Renato M.

2015-10-14 21:40 GMT+02:00 Bernardo Vecchia Stein <bernardovstein@gmail.com>
:

> Hi Renato,
>
> I have done that as well, but so far no luck. I believe spark is finding
> the library correctly, otherwise the error message would be "no libraryname
> found" or something like that. The problem seems to be something else, and
> I'm not sure how to find it.
>
> Thanks,
> Bernardo
>
> On 14 October 2015 at 16:28, Renato Marroquín Mogrovejo <
> renatoj.marroquin@gmail.com> wrote:
>
>> You can also try setting the env variable LD_LIBRARY_PATH to point where
>> your compiled libraries are.
>>
>>
>> Renato M.
>>
>> 2015-10-14 21:07 GMT+02:00 Bernardo Vecchia Stein <
>> bernardovstein@gmail.com>:
>>
>>> Hi Deenar,
>>>
>>> Yes, the native library is installed on all machines of the cluster. I
>>> tried a simpler approach by just using System.load() and passing the exact
>>> path of the library, and things still won't work (I get exactly the same
>>> error and message).
>>>
>>> Any ideas of what might be failing?
>>>
>>> Thank you,
>>> Bernardo
>>>
>>> On 14 October 2015 at 02:50, Deenar Toraskar <deenar.toraskar@gmail.com>
>>> wrote:
>>>
>>>> Hi Bernardo
>>>>
>>>> Is the native library installed on all machines of your cluster and are
>>>> you setting both the spark.driver.extraLibraryPath and
>>>> spark.executor.extraLibraryPath ?
>>>>
>>>> Deenar
>>>>
>>>>
>>>>
>>>> On 14 October 2015 at 05:44, Bernardo Vecchia Stein <
>>>> bernardovstein@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I am trying to run some scala code in cluster mode using spark-submit.
>>>>> This code uses addLibrary to link with a .so that exists in the machine,
>>>>> and this library has a function to be called natively (there's a native
>>>>> definition as needed in the code).
>>>>>
>>>>> The problem I'm facing is: whenever I try to run this code in cluster
>>>>> mode, spark fails with the following message when trying to execute the
>>>>> native function:
>>>>> java.lang.UnsatisfiedLinkError:
>>>>> org.name.othername.ClassName.nativeMethod([B[B)[B
>>>>>
>>>>> Apparently, the library is being found by spark, but the required
>>>>> function isn't found.
>>>>>
>>>>> When trying to run in client mode, however, this doesn't fail and
>>>>> everything works as expected.
>>>>>
>>>>> Does anybody have any idea of what might be the problem here? Is there
>>>>> any bug that could be related to this when running in cluster mode?
>>>>>
>>>>> I appreciate any help.
>>>>> Thanks,
>>>>> Bernardo
>>>>>
>>>>
>>>>
>>>
>>
>

Mime
View raw message