spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Nist <tsind...@gmail.com>
Subject Re: Using P4J Plugins with Spark
Date Tue, 21 Apr 2020 13:25:37 GMT
You may want to make sure you include the jar of P4J and your plugins as
part of the following so that both the driver and executors have access.
If HDFS is out then you could
make a common mount point on each of the executor nodes so they have access
to the classes.


   - spark-submit --jars /common/path/to/jars
   - spark.driver.extraClassPath or it's alias --driver-class-path to set
   extra classpaths on the node running the driver.
   - spark.executor.extraClassPath to set extra class path on the Worker
   nodes.


On Tue, Apr 21, 2020 at 1:13 AM Shashanka Balakuntala <
shbalakuntala@gmail.com> wrote:

> Hi users,
> I'm a bit of newbie to spark infrastructure. And i have a small doubt.
> I have a maven project with plugins generated separately in a folder and
> normal java command to run is as follows:
> `java -Dp4j.pluginsDir=./plugins -jar /path/to/jar`
>
> Now when I run this program in local with spark-submit with standalone
> cluster(not cluster mode) the program compiles and plugins are in "plugins"
> folder in the $SPARK_HOME and it is getting recognised.
> The same is not the case in cluster mode. It says the Extenstion point is
> not loaded. Please advise on how can i create a folder which can be shared
> among the workers in "plugin" folder.
>
> PS: HDFS is not an options as we dont have a different setup
>
> Thanks.
>
>
> *Regards*
>   Shashanka Balakuntala Srinivasa
>
>

Mime
View raw message