spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: custom spark app name in yarn-cluster mode
Date Fri, 12 Dec 2014 07:35:00 GMT
Hi Tomer,

In yarn-cluster mode, the application has already been submitted to YARN by
the time the SparkContext is created, so it's too late to set the app name
there.  I believe giving it with the --name property to spark-submit should
work.

-Sandy

On Thu, Dec 11, 2014 at 10:28 AM, Tomer Benyamini <tomer.ben@gmail.com>
wrote:
>
>
>
> On Thu, Dec 11, 2014 at 8:27 PM, Tomer Benyamini <tomer.ben@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm trying to set a custom spark app name when running a java spark app
>> in yarn-cluster mode.
>>
>>  SparkConf sparkConf = new SparkConf();
>>
>>  sparkConf.setMaster(System.getProperty("spark.master"));
>>
>>  sparkConf.setAppName("myCustomName");
>>
>>  sparkConf.set("spark.logConf", "true");
>>
>>  JavaSparkContext sc = new JavaSparkContext(sparkConf);
>>
>>
>> Apparently this only works when running in yarn-client mode; in
>> yarn-cluster mode the app name is the class name, when viewing the app in
>> the cluster manager UI. Any idea?
>>
>>
>> Thanks,
>>
>> Tomer
>>
>>
>>
>

Mime
View raw message