spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ping Liu <pingpinga...@gmail.com>
Subject Re: Is it feasible to build and run Spark on Windows?
Date Thu, 05 Dec 2019 22:51:50 GMT
Hi Deepak,

Yes, I did use Maven. I even have the build pass successfully when setting
Hadoop version to 3.2.  Please see my response to Sean's email.

Unfortunately, I only have Docker Toolbox as my Windows doesn't have
Microsoft Hyper-V.  So I want to avoid using Docker to do major work if
possible.

Thanks!

Ping


On Thu, Dec 5, 2019 at 2:24 PM Deepak Vohra <dvohra09@yahoo.com> wrote:

> Several alternatives are available:
>
> - Use Maven to build Spark on Windows.
> http://spark.apache.org/docs/latest/building-spark.html#apache-maven
>
> - Use Docker image for  CDH on Windows
> Docker Hub <https://hub.docker.com/u/cloudera>
>
> Docker Hub
>
> <https://hub.docker.com/u/cloudera>
>
>
>
>
> On Thursday, December 5, 2019, 09:33:43 p.m. UTC, Sean Owen <
> srowen@gmail.com> wrote:
>
>
> What was the build error? you didn't say. Are you sure it succeeded?
> Try running from the Spark home dir, not bin.
> I know we do run Windows tests and it appears to pass tests, etc.
>
> On Thu, Dec 5, 2019 at 3:28 PM Ping Liu <pingpinganan@gmail.com> wrote:
> >
> > Hello,
> >
> > I understand Spark is preferably built on Linux.  But I have a Windows
> machine with a slow Virtual Box for Linux.  So I wish I am able to build
> and run Spark code on Windows environment.
> >
> > Unfortunately,
> >
> > # Apache Hadoop 2.6.X
> > ./build/mvn -Pyarn -DskipTests clean package
> >
> > # Apache Hadoop 2.7.X and later
> > ./build/mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean
> package
> >
> >
> > Both are listed on
> http://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version-and-enabling-yarn
> >
> > But neither works for me (I stay directly under spark root directory and
> run "mvn -Pyarn -Phadoop-2.7 -Dhadoop.version=2.7.3 -DskipTests clean
> package"
> >
> > and
> >
> > Then I tried "mvn -Pyarn -Phadoop-3.2 -Dhadoop.version=3.2.1 -DskipTests
> clean package"
> >
> > Now build works.  But when I run spark-shell.  I got the following error.
> >
> > D:\apache\spark\bin>spark-shell
> > Exception in thread "main" java.lang.NoSuchMethodError:
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
> >        at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
> >        at
> org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
> >        at
> org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendS3AndSparkHadoopHiveConfigurations(SparkHadoopUtil.scala:456)
> >        at
> org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:427)
> >        at
> org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
> >        at
> org.apache.spark.deploy.SparkSubmit$$Lambda$132/817978763.apply(Unknown
> Source)
> >        at scala.Option.getOrElse(Option.scala:189)
> >        at
> org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
> >        at org.apache.spark.deploy.SparkSubmit.org
> $apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
> >        at
> org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
> >        at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> >        at
> org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
> >        at
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
> >        at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
> >        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> >
> >
> > Has anyone experienced building and running Spark source code
> successfully on Windows?  Could you please share your experience?
> >
> > Thanks a lot!
> >
> > Ping
>
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
>

Mime
View raw message