spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Starks <suse...@protonmail.com.INVALID>
Subject Pass config file through spark-submit
Date Thu, 16 Aug 2018 14:29:52 GMT
I have a config file that exploits type safe config library located on the local file system,
and want to submit that file through spark-submit so that spark program can read customized
parameters. For instance,

my.app {
  db {
    host = domain.cc
    port = 1234
    db = dbname
    user = myuser
    passwd = mypass
  }
}

Spark submit code looks like

    spark-submit --class "my.app.Sample" --master local[*] --conf "spark.executor.extraJavaOptions=-Dconfig.file=/path/to/conf/myapp.conf"
/path/to/my-app.jar

But the program can not read the parameters such as db, user, host, and so on in my conf file.

Passing with --files /path/to/myapp.conf doesn't work either.

What is the correct way to submit that kind of conf file so that my spark job can read customized
parameters from there?

Thanks
Mime
View raw message