spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Mistroni <mmistr...@gmail.com>
Subject Re: Pls assist: Spark 2.0 build failure on Ubuntu 16.06
Date Sat, 01 Oct 2016 20:24:19 GMT
Hi guys
 sorry to annoy you on this but i am getting nowhere. So far i have tried
to build spark 2.0 on my local laptop with no success.... so i blamed my
laptop poor performance
So today i fired off an EC2 Ubuntu 16.06 Instance and installed the
following (i copy paste commands here)

ubuntu@ip-172-31-40-104:~/spark$ history
    sudo apt-get install -y python-software-properties
    sudo apt-get install -y software-properties-common
    sudo add-apt-repository -y ppa:webupd8team/java
    sudo apt-get install -y oracle-java8-installer
    sudo apt-get install -y git
    git clone git://github.com/apache/spark.git
    cd spark

Then launched the following commands:
First thi sone, with only yarn, as i dont need hadoop

 ./build/mvn -Pyarn  -DskipTests clean package

This failed. after kicking off same command with -X i got this

ERROR] Failed to execute goal
net.alchim31.maven:scala-maven-plugin:3.2.2:compile (scala-compile-first)
on project spark-core_2.11: Execution scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed. CompileFailed
-> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile
(scala-compile-first) on project spark-core_2.11: Execution
scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.
        at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
        at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
        at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
        at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
        at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
        at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
        at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
        at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
        at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
        at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
        at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
        at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
        at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.PluginExecutionException: Execution
scala-compile-first of goal
net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.
        at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:145)
        at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
        ... 20 more
Caused by: Compile failed via zinc server
        at
sbt_inc.SbtIncrementalCompiler.zincCompile(SbtIncrementalCompiler.java:136)
        at
sbt_inc.SbtIncrementalCompiler.compile(SbtIncrementalCompiler.java:86)
        at
scala_maven.ScalaCompilerSupport.incrementalCompile(ScalaCompilerSupport.java:303)
        at
scala_maven.ScalaCompilerSupport.compile(ScalaCompilerSupport.java:119)
        at
scala_maven.ScalaCompilerSupport.doExecute(ScalaCompilerSupport.java:99)
        at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482)
        at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
        ... 21 more



Then i tried to use hadoop even though i dont need hadoop for my code

./build/mvn -Pyarn -Phadoop-2.4  -DskipTests clean package

This failed again, at exactly the same point, with the same error

Then i thought maybe i used an old version of hadoop, so i tried to use 2.7

  ./build/mvn -Pyarn -Phadoop-2.7  -DskipTests clean package

Nada. same error.


I have tried sbt, no luck

Could anyone suggest me a mvn proflie to use to build spark 2.0? i am
starting to suspect (before arguing that there's something wrong with spark
git) that i am using somehow wrong parameters..... or perhaps i should
install scala 2.11 before i install spark? or Maven  ?

kr
 marco


On Fri, Sep 30, 2016 at 8:23 PM, Marco Mistroni <mmistroni@gmail.com> wrote:

> Hi all
>  this problem is still bothering me.
> Here's my setup
> - Ubuntu 16.06
> - Java 8
> - Spark 2.0
> - have launched following command: ./build/mvn -X -Pyarn -Phadoop-2.7
> -DskipTests clean package
> and i am gettign this exception:
>
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile
> (scala-test-compile-first) on project spark-mllib_2.10: Execution
> scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile
> failed.
>
>         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:212)
>         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:153)
>         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:145)
>         at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.
> buildProject(LifecycleModuleBuilder.java:116)
>         at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.
> buildProject(LifecycleModuleBuilder.java:80)
>         at org.apache.maven.lifecycle.internal.builder.singlethreaded.
> SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
>         at org.apache.maven.lifecycle.internal.LifecycleStarter.
> execute(LifecycleStarter.java:128)
>         at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
>         at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
>         at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
>         at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
>         at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
>         at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.codehaus.plexus.classworlds.launcher.Launcher.
> launchEnhanced(Launcher.java:289)
>         at org.codehaus.plexus.classworlds.launcher.Launcher.
> launch(Launcher.java:229)
>         at org.codehaus.plexus.classworlds.launcher.Launcher.
> mainWithExitCode(Launcher.java:415)
>         at org.codehaus.plexus.classworlds.launcher.Launcher.
> main(Launcher.java:356)
> Caused by: org.apache.maven.plugin.PluginExecutionException: Execution
> scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile
> failed.
>         at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(
> DefaultBuildPluginManager.java:145)
>         at org.apache.maven.lifecycle.internal.MojoExecutor.execute(
> MojoExecutor.java:207)
>         ... 20 more
> Caused by: Compile failed via zinc server
>         at sbt_inc.SbtIncrementalCompiler.zincCompile(
> SbtIncrementalCompiler.java:136)
>         at sbt_inc.SbtIncrementalCompiler.compile(
> SbtIncrementalCompiler.java:86)
>         at scala_maven.ScalaCompilerSupport.incrementalCompile(
> ScalaCompilerSupport.java:303)
>         at scala_maven.ScalaCompilerSupport.compile(
> ScalaCompilerSupport.java:119)
>         at scala_maven.ScalaCompilerSupport.doExecute(
> ScalaCompilerSupport.java:99)
>         at scala_maven.ScalaMojoSupport.execute(ScalaMojoSupport.java:482)
>         at scala_maven.ScalaTestCompileMojo.execute(
> ScalaTestCompileMojo.java:48)
>         at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(
> DefaultBuildPluginManager.java:134)
>
> anyone found a similar error?
>
> kr
>
>
>
>
> On Sat, Sep 3, 2016 at 2:54 PM, Marco Mistroni <mmistroni@gmail.com>
> wrote:
>
>> hi all
>>
>>  i am getting failures when building spark 2.0 on Ubuntu 16.06
>> Here's details of what i have installed on the ubuntu host
>> -  java 8
>> - scala 2.11
>> - git
>>
>> When i launch the command
>>
>> ./build/mvn  -Pyarn -Phadoop-2.7  -DskipTests clean package
>>
>> everything compiles sort of fine and at the end i get this exception
>>
>> INFO] BUILD FAILURE
>> [INFO] ------------------------------------------------------------
>> ------------
>> [INFO] Total time: 01:05 h
>> [INFO] Finished at: 2016-09-03T13:25:27+00:00
>> [INFO] Final Memory: 57M/208M
>> [INFO] ------------------------------------------------------------
>> ------------
>> [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile
>> (scala-test-compile-first) on project spark-streaming_2.11: Execution
>> scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile
>> failed. CompileFailed -> [Help 1]
>> [ERROR]
>> [ERROR] To see the full stack trace of the errors, re-run Maven with the
>> -e switch.
>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>> [ERROR]
>> [ERROR] For more information about the errors and possible solutions,
>> please read the following articles:
>> [ERROR] [Help 1] http://cwiki.apache.org/conflu
>> ence/display/MAVEN/PluginExecutionException
>> [ERROR]
>> [ERROR] After correcting the problems, you can resume the build with the
>> command
>> [ERROR]   mvn <goals> -rf :spark-streaming_2.11
>>
>> could anyone help?
>>
>> kr
>>
>
>

Mime
View raw message