spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Rosen <rosenvi...@gmail.com>
Subject Re: Spark build/standalone on Windows 8.1
Date Sun, 15 Dec 2013 01:31:21 GMT
Are you running make-distribution.sh under the Cygwin bash shell or in a
DOS or PowerShell prompt?  When I tried it with the latter, the Windows 8
"choose an application to open this file" dialog appears rather than
running the script.

I received a different set of errors when running make-distribution.sh
under Cygwin:


joshrosen@joshrosen-PC/cygdrive/c/users/joshrosen/Documents/GitHub/incubator-spark
$ ./make-distribution.sh
Error: Unable to access jarfile
/cygdrive/c/users/joshrosen/Documents/GitHub/incubator-spark/sbt/sbt-launch-0.11.3-2.jar
Making distribution for Spark in
/cygdrive/c/users/joshrosen/Documents/GitHub/incubator-spark/dist...
Hadoop version set to 1.0.4
YARN disabled
Error: Unable to access jarfile
/cygdrive/c/users/joshrosen/Documents/GitHub/incubatorspark/sbt/sbt-launch-0.11.3-2.jar

I'm using the Oracle JDK.  I think my error is because the java
executable isn't prepared to handle the Unix jar paths passed from Cygwin.

I was able to successfully assemble Spark by opening a PowerShell prompt as
administrator and running "sbt\sbt.cmd assembly" in the Spark directory.


On Wed, Dec 4, 2013 at 2:13 PM, Adrian Bonar <Adrian.Bonar@microsoft.com>wrote:

>  I am trying to bring up a standalone cluster on Windows 8.1/Windows
> Server 2013 and I am having troubles getting the make-distribution script
> to complete successfully. It seems to change a bunch of permissions in
> /dist and then tries to write to it unsuccessfully. I assume this is not
> expected behavior.
>
>
>
> Any thoughts are greatly appreciated.
>
>
>
> Here is my workflow (and results):
>
>
>
>  Building/Making a Spark distribution
>
> 1.       Install Cygwin (64-bit)
>
> 2.       Install JDK 7u45
>
> a.       Add JDK to PATH (i.e. set PATH=%PATH%;C:\Program
> Files\Java\jdk1.7.0_45\bin)
>
> 3.       Install GIT 1.8.5.1
>
> 4.       Download Spark 0.8.0-inclubating
>
> 5.       Right-click on downloaded spark file, go to properties, and
> click on "Unblock"
>
> 6.       Extract to d:\spark
>
> 11.   Open a DOS command prompt as administrator
>
> 12.   c:\> *cd /d d:\spark*
>
> 13.   d:\spark> *make-distribution.sh*
>
> Welcome to Git (version 1.8.4-preview20130916)
>
>
>
>
>
> Run 'git help git' to display the help index.
>
> Run 'git help <command>' to display help for specific commands.
>
> Error: Could not create the Java Virtual Machine.
>
> Error: A fatal exception has occurred. Program will exit.
>
> Making distribution for Spark Could in /d/spark/dist...
>
> Hadoop version set to 1.0.4
>
> YARN disabled
>
> Error occurred during initialization of VM
>
> Could not reserve enough space for object heap
>
> Error: Could not create the Java Virtual Machine.
>
> Error: A fatal exception has occurred. Program will exit.
>
> rm: cannot change to directory `/d/spark/dist/bin': Permission denied
>
> rm: cannot change to directory `/d/spark/dist/python': Permission denied
>
> rm: cannot remove directory `/d/spark/dist': Directory not empty
>
> cp: cannot stat `/d/spark/assembly/target/scala*/*assembly*hadoop*.jar':
> No such file or directory
>
> cp: cannot stat `/d/spark/conf/*.template': No such file or directory
>
> cp: cannot create regular file `/d/spark/dist/bin/compute-classpath.cmd':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/bin/compute-classpath.sh':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/bin/slaves.sh': Permission
> denied
>
> cp: cannot create regular file `/d/spark/dist/bin/spark-config.sh':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/bin/spark-daemon.sh':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/bin/spark-daemons.sh':
> Permission denied
>
> .
>
> .
>
> .
>
> cp: cannot create regular file `/d/spark/dist/python/pyspark/__init__.py':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/python/run-tests':
> Permission denied
>
> cp: cannot create regular file
> `/d/spark/dist/python/test_support/hello.txt': Permission denied
>
> cp: cannot create regular file
> `/d/spark/dist/python/test_support/userlib-0.1-py2.7.egg': Permission denied
>
> cp: cannot create regular file
> `/d/spark/dist/python/test_support/userlibrary.py': Permission denied
>
>
>
> 14.   d:\spark> *sbt\sbt assembly*
>
> …
>
> [info] Compiling 10 Scala sources to
> D:\spark\repl\target\scala-2.9.3\classes...
>
> [info] Compiling 39 Scala sources and 13 Java sources to
> D:\spark\examples\target\scala-2.9.3\classes...
>
> sbt appears to be exiting abnormally.
>
>   The log file for this session is at
> C:\Users\adribona\AppData\Local\Temp\sbt7403459132156612750.log
>
> java.lang.OutOfMemoryError: PermGen space
>
>         at java.util.concurrent.FutureTask.report(Unknown Source)
>
>         at java.util.concurrent.FutureTask.get(Unknown Source)
>
>         at
> sbt.ConcurrentRestrictions$$anon$4.take(ConcurrentRestrictions.scala:196)
>
>         at sbt.Execute.next$1(Execute.scala:85)
>
>         at sbt.Execute.processAll(Execute.scala:88)
>
>         at sbt.Execute.runKeep(Execute.scala:68)
>
>         at sbt.EvaluateTask$.run$1(EvaluateTask.scala:162)
>
>         at sbt.EvaluateTask$.runTask(EvaluateTask.scala:177)
>
>         at sbt.Aggregation$$anonfun$4.apply(Aggregation.scala:46)
>
>         at sbt.Aggregation$$anonfun$4.apply(Aggregation.scala:44)
>
>         at sbt.EvaluateTask$.withStreams(EvaluateTask.scala:137)
>
>         at sbt.Aggregation$.runTasksWithResult(Aggregation.scala:44)
>
>         at sbt.Aggregation$.runTasks(Aggregation.scala:59)
>
>         at
> sbt.Aggregation$$anonfun$applyTasks$1.apply(Aggregation.scala:31)
>
>         at
> sbt.Aggregation$$anonfun$applyTasks$1.apply(Aggregation.scala:30)
>
>         at
> sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
>
>         at
> sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
>
>         at sbt.Command$.process(Command.scala:90)
>
>         at
> sbt.MainLoop$$anonfun$next$1$$anonfun$apply$1.apply(MainLoop.scala:71)
>
>         at
> sbt.MainLoop$$anonfun$next$1$$anonfun$apply$1.apply(MainLoop.scala:71)
>
>         at sbt.State$$anon$2.process(State.scala:171)
>
>         at sbt.MainLoop$$anonfun$next$1.apply(MainLoop.scala:71)
>
>         at sbt.MainLoop$$anonfun$next$1.apply(MainLoop.scala:71)
>
>         at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
>
>         at sbt.MainLoop$.next(MainLoop.scala:71)
>
>         at sbt.MainLoop$.run(MainLoop.scala:64)
>
>         at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:53)
>
>         at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:50)
>
>         at sbt.Using.apply(Using.scala:25)
>
>         at sbt.MainLoop$.runWithNewLog(MainLoop.scala:50)
>
>         at sbt.MainLoop$.runAndClearLast(MainLoop.scala:33)
>
>         at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:17)
>
> Error during sbt execution: java.lang.OutOfMemoryError: PermGen space
>
>
>
> 15.   d:\spark> *set _JAVA_OPTIONS="-Xmx512M"*
>
>
>
> 16.   d:\spark>* sbt\sbt assembly*
>
> …
>
> [info] SHA-1: 60c513d9b066c9e7a9b51d6282a8e4f9f55b62e6
>
> [info] Packaging
> D:\spark\examples\target\scala-2.9.3\spark-examples-assembly-0.8.0-incubating.jar
> ...
>
> [info] Done packaging.
>
> [success] Total time: 319 s, completed Dec 4, 2013 12:44:18 PM
>
>
>
> 17.   d:\spark> *make-distribution.sh*
>
> …
>
> [warn] Strategy 'concat' was applied to a file
>
> [warn] Strategy 'discard' was applied to 2 files
>
> [warn] Strategy 'first' was applied to 212 files
>
> [info] Checking every *.class/*.jar file's SHA-1.
>
> [info] Assembly up to date:
> D:\spark\assembly\target\scala-2.9.3\spark-assembly-0.8.0-incubating-hadoop1.0.4.jar
>
> [success] Total time: 312 s, completed Dec 4, 2013 1:02:02 PM
>
> rm: cannot change to directory `/d/spark/dist/bin': Permission denied
>
> rm: cannot change to directory `/d/spark/dist/python': Permission denied
>
> rm: cannot remove directory `/d/spark/dist': Directory not empty
>
> cp: cannot stat `/d/spark/conf/*.template': No such file or directory
>
> cp: cannot create regular file `/d/spark/dist/bin/compute-classpath.cmd':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/bin/compute-classpath.sh':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/bin/slaves.sh': Permission
> denied
>
> cp: cannot create regular file `/d/spark/dist/bin/spark-config.sh':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/bin/spark-daemon.sh':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/bin/spark-daemons.sh':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/bin/start-all.sh':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/bin/start-master.sh':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/bin/start-slave.sh':
> Permission denied
>
> .
>
> .
>
> .
>
> cp: cannot create regular file `/d/spark/dist/python/pyspark/worker.py':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/python/pyspark/__init__.py':
> Permission denied
>
> cp: cannot create regular file `/d/spark/dist/python/run-tests':
> Permission denied
>
> cp: cannot create regular file
> `/d/spark/dist/python/test_support/hello.txt': Permission denied
>
> cp: cannot create regular file
> `/d/spark/dist/python/test_support/userlib-0.1-py2.7.egg': Permission denied
>
> cp: cannot create regular file
> `/d/spark/dist/python/test_support/userlibrary.py': Permission denied
>
>
>
> 18.   d:\spark> *cd dist\bin*
>
> Access is denied.
>
>
>
> --Adrian
>

Mime
View raw message