spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bernardo Vecchia Stein <bernardovst...@gmail.com>
Subject Re: Running in cluster mode causes native library linking to fail
Date Wed, 14 Oct 2015 19:07:28 GMT
Hi Deenar,

Yes, the native library is installed on all machines of the cluster. I
tried a simpler approach by just using System.load() and passing the exact
path of the library, and things still won't work (I get exactly the same
error and message).

Any ideas of what might be failing?

Thank you,
Bernardo

On 14 October 2015 at 02:50, Deenar Toraskar <deenar.toraskar@gmail.com>
wrote:

> Hi Bernardo
>
> Is the native library installed on all machines of your cluster and are
> you setting both the spark.driver.extraLibraryPath and
> spark.executor.extraLibraryPath ?
>
> Deenar
>
>
>
> On 14 October 2015 at 05:44, Bernardo Vecchia Stein <
> bernardovstein@gmail.com> wrote:
>
>> Hello,
>>
>> I am trying to run some scala code in cluster mode using spark-submit.
>> This code uses addLibrary to link with a .so that exists in the machine,
>> and this library has a function to be called natively (there's a native
>> definition as needed in the code).
>>
>> The problem I'm facing is: whenever I try to run this code in cluster
>> mode, spark fails with the following message when trying to execute the
>> native function:
>> java.lang.UnsatisfiedLinkError:
>> org.name.othername.ClassName.nativeMethod([B[B)[B
>>
>> Apparently, the library is being found by spark, but the required
>> function isn't found.
>>
>> When trying to run in client mode, however, this doesn't fail and
>> everything works as expected.
>>
>> Does anybody have any idea of what might be the problem here? Is there
>> any bug that could be related to this when running in cluster mode?
>>
>> I appreciate any help.
>> Thanks,
>> Bernardo
>>
>
>

Mime
View raw message