spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Theodore Wong <...@tmwong.org>
Subject Re: How to pass config variables to workers
Date Fri, 16 May 2014 22:16:01 GMT
I found that the easiest way was to pass variables in the Spark configuration
object. The only catch is that all of your properties keys must being with
"spark." in order for Spark to propagate the values. So, for example, in the
driver:

SparkConf conf = new SparkConf();
conf.set("spark.myapp.myproperty", "propertyValue");

JavaSparkContext context = new JavaSparkContext(conf);

I realize that this is most likely a hack, but it works and is easy (at
least for me) to follow from a programming standpoint compared to setting
environment variables outside of the program.

Regards,

Theodore Wong



-----
-- 
Theodore Wong &lt;tmw@tmwong.org&gt;
www.tmwong.org

--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-to-pass-config-variables-to-workers-tp5780p5880.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message