spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: NO Cygwin Support in bin/spark-class in Spark 1.4.0
Date Wed, 29 Jul 2015 02:34:28 GMT
Can you run the windows batch files (e.g. spark-submit.cmd) from the cygwin
shell?

On Tue, Jul 28, 2015 at 7:26 PM, Proust GZ Feng <pfeng@cn.ibm.com> wrote:

> Hi, Owen
>
> Add back the cygwin classpath detection can pass the issue mentioned
> before, but there seems lack of further support in the launch lib, see
> below stacktrace
>
> LAUNCH_CLASSPATH:
> C:\spark-1.4.0-bin-hadoop2.3\lib\spark-assembly-1.4.0-hadoop2.3.0.jar
> java -cp
> *C:\spark-1.4.0-bin-hadoop2.3\lib\spark-assembly-1.4.0-hadoop2.3.0.jar*
> org.apache.spark.launcher.Main org.apache.spark.deploy.SparkSubmit
> --driver-class-path ../thirdparty/lib/db2-jdbc4-95fp6a/db2jcc4.jar
> --properties-file conf/spark.properties
> target/scala-2.10/price-scala-assembly-15.4.0-SNAPSHOT.jar
> Exception in thread "main" java.lang.IllegalStateException: Library
> directory '*C:\c\spark-1.4.0-bin-hadoop2.3\lib_managed\jars*' does not
> exist.
>         at
> org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:229)
>         at
> org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:215)
>         at
> org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:115)
>         at
> org.apache.spark.launcher.SparkSubmitCommandBuilder.buildSparkSubmitCommand(SparkSubmitCommandBuilder.java:192)
>         at
> org.apache.spark.launcher.SparkSubmitCommandBuilder.buildCommand(SparkSubmitCommandBuilder.java:117)
>         at org.apache.spark.launcher.Main.main(Main.java:74)
>
> Thanks
> Proust
>
>
>
>
> From:        Sean Owen <sowen@cloudera.com>
> To:        Proust GZ Feng/China/IBM@IBMCN
> Cc:        user <user@spark.apache.org>
> Date:        07/28/2015 06:54 PM
> Subject:        Re: NO Cygwin Support in bin/spark-class in Spark 1.4.0
> ------------------------------
>
>
>
> Does adding back the cygwin detection and this clause make it work?
>
> if $cygwin; then
>  CLASSPATH="`cygpath -wp "$CLASSPATH"`"
> fi
>
> If so I imagine that's fine to bring back, if that's still needed.
>
> On Tue, Jul 28, 2015 at 9:49 AM, Proust GZ Feng <pfeng@cn.ibm.com> wrote:
> > Thanks Owen, the problem under Cygwin is while run spark-submit under
> 1.4.0,
> > it simply report
> >
> > Error: Could not find or load main class org.apache.spark.launcher.Main
> >
> > This is because under Cygwin spark-class make the LAUNCH_CLASSPATH as
> > "/c/spark-1.4.0-bin-hadoop2.3/lib/spark-assembly-1.4.0-hadoop2.3.0.jar"
> > But under Cygwin java in Windows cannot recognize the classpath, so below
> > command simply error out
> >
> >  java -cp
> > /c/spark-1.4.0-bin-hadoop2.3/lib/spark-assembly-1.4.0-hadoop2.3.0.jar
> > org.apache.spark.launcher.Main
> > Error: Could not find or load main class org.apache.spark.launcher.Main
> >
> > Thanks
> > Proust
> >
> >
> >
> > From:        Sean Owen <sowen@cloudera.com>
> > To:        Proust GZ Feng/China/IBM@IBMCN
> > Cc:        user <user@spark.apache.org>
> > Date:        07/28/2015 02:20 PM
> > Subject:        Re: NO Cygwin Support in bin/spark-class in Spark 1.4.0
> > ________________________________
> >
> >
> >
> > It wasn't removed, but rewritten. Cygwin is just a distribution of
> > POSIX-related utilities so you should be able to use the normal .sh
> > scripts. In any event, you didn't say what the problem is?
> >
> > On Tue, Jul 28, 2015 at 5:19 AM, Proust GZ Feng <pfeng@cn.ibm.com>
> wrote:
> >> Hi, Spark Users
> >>
> >> Looks like Spark 1.4.0 cannot work with Cygwin due to the removing of
> >> Cygwin
> >> support in bin/spark-class
> >>
> >> The changeset is
> >>
> >>
> https://github.com/apache/spark/commit/517975d89d40a77c7186f488547eed11f79c1e97#diff-fdf4d3e600042c63ffa17b692c4372a3
> >>
> >> The changeset said "Add a library for launching Spark jobs
> >> programmatically", but how to use it in Cygwin?
> >> I'm wondering any solutions available to make it work in Windows?
> >>
> >>
> >> Thanks
> >> Proust
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
>


-- 
Marcelo

Mime
View raw message