spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Odersky <ja...@odersky.com>
Subject Re: How to distribute dependent files (.so , jar ) across spark worker nodes
Date Mon, 14 Mar 2016 21:31:35 GMT
Have you tried setting the configuration
`spark.executor.extraLibraryPath` to point to a location where your
.so's are available? (Not sure if non-local files, such as HDFS, are
supported)

On Mon, Mar 14, 2016 at 2:12 PM, Tristan Nixon <stuff@memeticlabs.org> wrote:
> What build system are you using to compile your code?
> If you use a dependency management system like maven or sbt, then you should be able
to instruct it to build a single jar that contains all the other dependencies, including third-party
jars and .so’s. I am a maven user myself, and I use the shade plugin for this:
> https://maven.apache.org/plugins/maven-shade-plugin/
>
> However, if you are using SBT or another dependency manager, someone else on this list
may be able to give you help on that.
>
> If you’re not using a dependency manager - well, you should be. Trying to manage this
manually is a pain that you do not want to get in the way of your project. There are perfectly
good tools to do this for you; use them.
>
>> On Mar 14, 2016, at 3:56 PM, prateek arora <prateek.arora2k6@gmail.com> wrote:
>>
>> Hi
>>
>> Thanks for the information .
>>
>> but my problem is that if i want to write spark application which depend on
>> third party libraries like opencv then whats is the best approach to
>> distribute all .so and jar file of opencv in all cluster ?
>>
>> Regards
>> Prateek
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-distribute-dependent-files-so-jar-across-spark-worker-nodes-tp26464p26489.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message