spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Lewis <lordjoe2...@gmail.com>
Subject Questions about serialization and SparkConf
Date Wed, 29 Oct 2014 18:57:09 GMT
     Assume in my executor I say

SparkConf sparkConf = new SparkConf();
        sparkConf.set("spark.kryo.registrator",
"com.lordjoe.distributed.hydra.HydraKryoSerializer");
       sparkConf.set("mysparc.data", "Some user Data");
         sparkConf.setAppName("Some App");

Now
   1) Are there default values set in some system file which are populated
if I call new SparkConf - if not how do I get those? _ I think i see
defaults foe the master, the Serializer...
    2) If I set a property in SparkConf for my SparkContext will I see that
property in a Slave machine?
     3) If I set a property anf then call showSparkProperties() do I see
that property set and if not how can I see the property set - say in
another thread as in
 if in some other thread on the executor   say as in
showSparkPropertiesInAnotherThread();
      4) How can a slave machine access properties set on the executor

I an really interested in   sparkConf.set("spark.kryo.registrator",
"com.lordjoe.distributed.hydra.HydraKryoSerializer");
    which needs to be used by the Slave


   /**
     * dump all spark properties to System.err
     */
    public static void showSparkProperties()
    {
        SparkConf sparkConf = new SparkConf();
        Tuple2<String, String>[] all = sparkConf.getAll();
        for (Tuple2<String, String> prp  : all) {
            System.err.println(prp._1().toString() + "=" + prp._2());
        }
    }

    public static void  showSparkPropertiesInAnotherThread()
    {
        new Thread(new Runnable() {
            @Override
            public void run() {
                showSparkProperties();
            }
        }).start();
    }

Mime
View raw message