spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Lewis <>
Subject Questions about serialization and SparkConf
Date Wed, 29 Oct 2014 18:57:09 GMT
     Assume in my executor I say

SparkConf sparkConf = new SparkConf();
       sparkConf.set("", "Some user Data");
         sparkConf.setAppName("Some App");

   1) Are there default values set in some system file which are populated
if I call new SparkConf - if not how do I get those? _ I think i see
defaults foe the master, the Serializer...
    2) If I set a property in SparkConf for my SparkContext will I see that
property in a Slave machine?
     3) If I set a property anf then call showSparkProperties() do I see
that property set and if not how can I see the property set - say in
another thread as in
 if in some other thread on the executor   say as in
      4) How can a slave machine access properties set on the executor

I an really interested in   sparkConf.set("spark.kryo.registrator",
    which needs to be used by the Slave

     * dump all spark properties to System.err
    public static void showSparkProperties()
        SparkConf sparkConf = new SparkConf();
        Tuple2<String, String>[] all = sparkConf.getAll();
        for (Tuple2<String, String> prp  : all) {
            System.err.println(prp._1().toString() + "=" + prp._2());

    public static void  showSparkPropertiesInAnotherThread()
        new Thread(new Runnable() {
            public void run() {

View raw message