spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Bonar <Adrian.Bo...@microsoft.com>
Subject Spark build/standalone on Windows 8.1
Date Wed, 04 Dec 2013 22:13:31 GMT
I am trying to bring up a standalone cluster on Windows 8.1/Windows Server 2013 and I am having
troubles getting the make-distribution script to complete successfully. It seems to change
a bunch of permissions in /dist and then tries to write to it unsuccessfully. I assume this
is not expected behavior.

Any thoughts are greatly appreciated.

Here is my workflow (and results):


 Building/Making a Spark distribution
1.       Install Cygwin (64-bit)
2.       Install JDK 7u45
a.       Add JDK to PATH (i.e. set PATH=%PATH%;C:\Program Files\Java\jdk1.7.0_45\bin)
3.       Install GIT 1.8.5.1
4.       Download Spark 0.8.0-inclubating
5.       Right-click on downloaded spark file, go to properties, and click on "Unblock"
6.       Extract to d:\spark
11.   Open a DOS command prompt as administrator
12.   c:\> cd /d d:\spark
13.   d:\spark> make-distribution.sh
Welcome to Git (version 1.8.4-preview20130916)


Run 'git help git' to display the help index.
Run 'git help <command>' to display help for specific commands.
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Making distribution for Spark Could in /d/spark/dist...
Hadoop version set to 1.0.4
YARN disabled
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
rm: cannot change to directory `/d/spark/dist/bin': Permission denied
rm: cannot change to directory `/d/spark/dist/python': Permission denied
rm: cannot remove directory `/d/spark/dist': Directory not empty
cp: cannot stat `/d/spark/assembly/target/scala*/*assembly*hadoop*.jar': No such file or directory
cp: cannot stat `/d/spark/conf/*.template': No such file or directory
cp: cannot create regular file `/d/spark/dist/bin/compute-classpath.cmd': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/compute-classpath.sh': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/slaves.sh': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/spark-config.sh': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/spark-daemon.sh': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/spark-daemons.sh': Permission denied
.
.
.
cp: cannot create regular file `/d/spark/dist/python/pyspark/__init__.py': Permission denied
cp: cannot create regular file `/d/spark/dist/python/run-tests': Permission denied
cp: cannot create regular file `/d/spark/dist/python/test_support/hello.txt': Permission denied
cp: cannot create regular file `/d/spark/dist/python/test_support/userlib-0.1-py2.7.egg':
Permission denied
cp: cannot create regular file `/d/spark/dist/python/test_support/userlibrary.py': Permission
denied

14.   d:\spark> sbt\sbt assembly
...
[info] Compiling 10 Scala sources to D:\spark\repl\target\scala-2.9.3\classes...
[info] Compiling 39 Scala sources and 13 Java sources to D:\spark\examples\target\scala-2.9.3\classes...
sbt appears to be exiting abnormally.
  The log file for this session is at C:\Users\adribona\AppData\Local\Temp\sbt7403459132156612750.log
java.lang.OutOfMemoryError: PermGen space
        at java.util.concurrent.FutureTask.report(Unknown Source)
        at java.util.concurrent.FutureTask.get(Unknown Source)
        at sbt.ConcurrentRestrictions$$anon$4.take(ConcurrentRestrictions.scala:196)
        at sbt.Execute.next$1(Execute.scala:85)
        at sbt.Execute.processAll(Execute.scala:88)
        at sbt.Execute.runKeep(Execute.scala:68)
        at sbt.EvaluateTask$.run$1(EvaluateTask.scala:162)
        at sbt.EvaluateTask$.runTask(EvaluateTask.scala:177)
        at sbt.Aggregation$$anonfun$4.apply(Aggregation.scala:46)
        at sbt.Aggregation$$anonfun$4.apply(Aggregation.scala:44)
        at sbt.EvaluateTask$.withStreams(EvaluateTask.scala:137)
        at sbt.Aggregation$.runTasksWithResult(Aggregation.scala:44)
        at sbt.Aggregation$.runTasks(Aggregation.scala:59)
        at sbt.Aggregation$$anonfun$applyTasks$1.apply(Aggregation.scala:31)
        at sbt.Aggregation$$anonfun$applyTasks$1.apply(Aggregation.scala:30)
        at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
        at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
        at sbt.Command$.process(Command.scala:90)
        at sbt.MainLoop$$anonfun$next$1$$anonfun$apply$1.apply(MainLoop.scala:71)
        at sbt.MainLoop$$anonfun$next$1$$anonfun$apply$1.apply(MainLoop.scala:71)
        at sbt.State$$anon$2.process(State.scala:171)
        at sbt.MainLoop$$anonfun$next$1.apply(MainLoop.scala:71)
        at sbt.MainLoop$$anonfun$next$1.apply(MainLoop.scala:71)
        at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
        at sbt.MainLoop$.next(MainLoop.scala:71)
        at sbt.MainLoop$.run(MainLoop.scala:64)
        at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:53)
        at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:50)
        at sbt.Using.apply(Using.scala:25)
        at sbt.MainLoop$.runWithNewLog(MainLoop.scala:50)
        at sbt.MainLoop$.runAndClearLast(MainLoop.scala:33)
        at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:17)
Error during sbt execution: java.lang.OutOfMemoryError: PermGen space

15.   d:\spark> set _JAVA_OPTIONS="-Xmx512M"

16.   d:\spark> sbt\sbt assembly
...
[info] SHA-1: 60c513d9b066c9e7a9b51d6282a8e4f9f55b62e6
[info] Packaging D:\spark\examples\target\scala-2.9.3\spark-examples-assembly-0.8.0-incubating.jar
...
[info] Done packaging.
[success] Total time: 319 s, completed Dec 4, 2013 12:44:18 PM

17.   d:\spark> make-distribution.sh
...
[warn] Strategy 'concat' was applied to a file
[warn] Strategy 'discard' was applied to 2 files
[warn] Strategy 'first' was applied to 212 files
[info] Checking every *.class/*.jar file's SHA-1.
[info] Assembly up to date: D:\spark\assembly\target\scala-2.9.3\spark-assembly-0.8.0-incubating-hadoop1.0.4.jar
[success] Total time: 312 s, completed Dec 4, 2013 1:02:02 PM
rm: cannot change to directory `/d/spark/dist/bin': Permission denied
rm: cannot change to directory `/d/spark/dist/python': Permission denied
rm: cannot remove directory `/d/spark/dist': Directory not empty
cp: cannot stat `/d/spark/conf/*.template': No such file or directory
cp: cannot create regular file `/d/spark/dist/bin/compute-classpath.cmd': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/compute-classpath.sh': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/slaves.sh': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/spark-config.sh': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/spark-daemon.sh': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/spark-daemons.sh': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/start-all.sh': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/start-master.sh': Permission denied
cp: cannot create regular file `/d/spark/dist/bin/start-slave.sh': Permission denied
.
.
.
cp: cannot create regular file `/d/spark/dist/python/pyspark/worker.py': Permission denied
cp: cannot create regular file `/d/spark/dist/python/pyspark/__init__.py': Permission denied
cp: cannot create regular file `/d/spark/dist/python/run-tests': Permission denied
cp: cannot create regular file `/d/spark/dist/python/test_support/hello.txt': Permission denied
cp: cannot create regular file `/d/spark/dist/python/test_support/userlib-0.1-py2.7.egg':
Permission denied
cp: cannot create regular file `/d/spark/dist/python/test_support/userlibrary.py': Permission
denied

18.   d:\spark> cd dist\bin
Access is denied.

--Adrian

Mime
View raw message