spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Starks <>
Subject Pass config file through spark-submit
Date Thu, 16 Aug 2018 14:29:52 GMT
I have a config file that exploits type safe config library located on the local file system,
and want to submit that file through spark-submit so that spark program can read customized
parameters. For instance, {
  db {
    host =
    port = 1234
    db = dbname
    user = myuser
    passwd = mypass

Spark submit code looks like

    spark-submit --class "" --master local[*] --conf "spark.executor.extraJavaOptions=-Dconfig.file=/path/to/conf/myapp.conf"

But the program can not read the parameters such as db, user, host, and so on in my conf file.

Passing with --files /path/to/myapp.conf doesn't work either.

What is the correct way to submit that kind of conf file so that my spark job can read customized
parameters from there?

View raw message