From user-return-74745-apmail-spark-user-archive=spark.apache.org@spark.apache.org Wed May 2 13:38:47 2018 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E7A8E18C56 for ; Wed, 2 May 2018 13:38:46 +0000 (UTC) Received: (qmail 94787 invoked by uid 500); 2 May 2018 13:38:41 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 94664 invoked by uid 500); 2 May 2018 13:38:41 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 94654 invoked by uid 99); 2 May 2018 13:38:41 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 02 May 2018 13:38:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id B1B391A049E for ; Wed, 2 May 2018 13:38:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.899 X-Spam-Level: * X-Spam-Status: No, score=1.899 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=datadoghq.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 53v_XFHVZKI9 for ; Wed, 2 May 2018 13:38:39 +0000 (UTC) Received: from mail-lf0-f43.google.com (mail-lf0-f43.google.com [209.85.215.43]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 16B305F124 for ; Wed, 2 May 2018 13:38:39 +0000 (UTC) Received: by mail-lf0-f43.google.com with SMTP id y72-v6so7163407lfd.2 for ; Wed, 02 May 2018 06:38:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datadoghq.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=i1k1JZVJ9MRG+lGkCMWOCPumG7jhLMKZeyOwt2vBK5c=; b=Af1Me3mHQVcyrr/EqdNYuR72CGOUi3Vqw68lPgio1rc1toON2gLh9OHLIaiJQ95Yxg Hu+u0AvrB+5mKt+vcBBSzXtZk+PX6s7M8b5wspnqF2/dI10KEcnh2MyA3SjpVsdicIpI qPJNtp4ifcm03qzHYFW0xganAG9ja9o0fn1qY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=i1k1JZVJ9MRG+lGkCMWOCPumG7jhLMKZeyOwt2vBK5c=; b=cTd4vHXZwh/fZ24SqjRErp8gSJ0UCIXE1WeM4DgUZJt5LUiGm/aTUy5/rxzddGPgqb Zl3RrNCjR8YqXWQThih9vcUHWKxtPr/9abtz18rVCMCj8AUEpfV1UQqiOtSaLn3rEjRM p2BWdGHprR6Pg1jr5f2nCTDAhrNnAmimnIb/AkUJ1x9plcYaTYoRSbKN9X7f8xuouxp+ CUI54mTS87OdOuzGFJZbPxIkHxYefpW9bliJJLLO3NIepSXgtpSoZaTG5C3x1WrPAztU pNZQrCc7AsI935lV/Ltg/T8MMaNmp79ftayQt5vZI9UD3jVdZS1NihlvBNokzuLon/K0 fPrw== X-Gm-Message-State: ALQs6tBdJg/nB4iJortu91EjSJQ2IETUVUEpKteWiV0d5LHhKh8988pD v6qsHU/dq+TrnmymOgJYJey8SPSQa8V8VkLBjQkEJg== X-Google-Smtp-Source: AB8JxZpyjXa4n2a26kd2QRtoJQvoYiyVtT1iUawSX6c5hdESEuJuSxDBmnlc90E877xxTZ5qucDbt42ObLg7Nv+dZGs= X-Received: by 2002:a19:3848:: with SMTP id d8-v6mr10172924lfj.16.1525268317744; Wed, 02 May 2018 06:38:37 -0700 (PDT) MIME-Version: 1.0 Received: by 10.46.159.73 with HTTP; Wed, 2 May 2018 06:38:37 -0700 (PDT) In-Reply-To: References: From: Vadim Semenov Date: Wed, 2 May 2018 09:38:37 -0400 Message-ID: Subject: Re: spark.executor.extraJavaOptions inside application code To: Agostino Calamita Cc: user Content-Type: multipart/alternative; boundary="000000000000444978056b3934db" --000000000000444978056b3934db Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable You need to pass config before creating a session val conf =3D new SparkConf() // All three methods below are equivalent conf.set("spark.executor.extraJavaOptions", "-Dbasicauth=3Dmyuser:mypasswor= d") conf.set("spark.executorEnv.basicauth", "myuser:mypassword") conf.setExecutorEnv("basicauth", "myuser:mypassword") val spark =3D SparkSession.builder().config(conf).appName("=E2=80=A6").getO= rCreate() On Wed, May 2, 2018 at 6:59 AM, Agostino Calamita < agostino.calamita@gmail.com> wrote: > Hi all, > I wrote an application that needs an environment variable. I can set this > variable with > > --conf 'spark.executor.extraJavaOptions=3D-Dbasicauth=3Dmyuser:mypwd' > > in spark-submit and it works well in standalone cluster mode. > > But, I want set it inside the application code, because the variable > contains a password. > > How can I do ? > > I tried with: > > SparkSession spark =3D SparkSession > .builder() > .appName("Java Spark Solr ETL") > .getOrCreate(); > > spark.sparkContext().conf().setExecutorEnv("spark.executor.extraJ= avaOptions", > "-Dbasicauth=3Dmyuser:mypassword"); > > but it doesn't work. > > Thanks. > --=20 Sent from my iPhone --000000000000444978056b3934db Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
You need to pass config before creating a session

= 
val conf =3D new SparkConf()
// All three methods below are equivalent
conf.=
set("spark.executor.extraJavaOptions", "-Dbasicauth=3Dmyuser:mypassword")
conf.set("spark.executorEnv.basicauth", "myuser:mypassword")conf.setExecutorEnv("basicauth", "myuser:mypassword")
val spark =3D SparkSess=
ion.builder=
().config(conf).appName("=E2=80=A6").getOr=
Create()

On Wed, May 2, 2018 at 6:59 AM, Agostino Calamita <agostino.calamita@gmail.com> wrote:
Hi all,
I wrote an application= that needs an environment variable. I can set this variable with
=

--conf 'spark.executor.extraJavaOptions=3D-Dbasicaut= h=3Dmyuser:mypwd'

in spark-submit and it works = well in standalone cluster mode.

But, I want set it insid= e the application code, because the variable contains a password.

How can I do ?

I tried with:

=C2=A0=C2= =A0=C2=A0 SparkSession spark =3D SparkSession
=C2=A0=C2=A0=C2=A0 =C2=A0= =C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0 .builder()
=C2= =A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 = =C2=A0 .appName("Java Spark Solr ETL")
=C2=A0=C2=A0=C2=A0 =C2= =A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 =C2=A0 .getOrCreate()= ;
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0
=C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 spark.sparkContext().conf().setExecutorEnv("spark.= executor.extraJavaOptions", "-Dbasicauth=3Dmyuser:mypassword= ");
=C2=A0=C2=A0=C2=A0
but it doesn't work.
<= br>
Thanks.



--
Se= nt from my iPhone
--000000000000444978056b3934db--