spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Evans <jeffrey.wayne.ev...@gmail.com>
Subject Re: Is it possible to obtain the full command to be invoked by SparkLauncher?
Date Thu, 25 Apr 2019 02:01:55 GMT
Thanks for the pointers.  We figured out the stdout/stderr capture
piece.  I was just looking to capture the full command in order to
help debug issues we run into with the submit depending on various
combinations of all parameters/classpath, and also to isolate job
specific issues from our wrapping application (i.e. being able to
submit the job directly, rather than through our app).  I will use the
environment variable method for now.

On Wed, Apr 24, 2019 at 4:18 PM Marcelo Vanzin <vanzin@cloudera.com> wrote:
>
> BTW the SparkLauncher API has hooks to capture the stderr of the
> spark-submit process into the logging system of the parent process.
> Check the API javadocs since it's been forever since I looked at that.
>
> On Wed, Apr 24, 2019 at 1:58 PM Marcelo Vanzin <vanzin@cloudera.com> wrote:
> >
> > Setting the SPARK_PRINT_LAUNCH_COMMAND env variable to 1 in the
> > launcher env will make Spark code print the command to stderr. Not
> > optimal but I think it's the only current option.
> >
> > On Wed, Apr 24, 2019 at 1:55 PM Jeff Evans
> > <jeffrey.wayne.evans@gmail.com> wrote:
> > >
> > > The org.apache.spark.launcher.SparkLauncher is used to construct a
> > > spark-submit invocation programmatically, via a builder pattern.  In
> > > our application, which uses a SparkLauncher internally, I would like
> > > to log the full spark-submit command that it will invoke to our log
> > > file, in order to aid in debugging/support.  However, I can't figure
> > > out a way to do this.  This snippet would work, except for the fact
> > > that the createBuilder method is private.
> > >
> > > sparkLauncher.createBuilder().command()
> > >
> > > Is there an alternate way of doing this?  The Spark version is
> > > 2.11:2.4.0.  Thanks.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe e-mail: user-unsubscribe@spark.apache.org
> > >
> >
> >
> > --
> > Marcelo
>
>
>
> --
> Marcelo

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org


Mime
View raw message