spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: spark config
Date Fri, 07 Aug 2015 22:19:59 GMT
In master branch, build/sbt-launch-lib.bash has the following:

  URL1=
https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar

I verified that the following exists:

https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.7/#sbt-launch.jar

FYI

On Fri, Aug 7, 2015 at 2:08 PM, Bryce Lobdell <lobdellb@gmail.com> wrote:

>
> I Recently downloaded spark package 1.4.0:
>
> A build of Spark with "sbt/sbt clean assembly" failed with message "Error:
> Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar"
>
> Upon investigation I figured out that "sbt-launch-0.13.7.jar" is
> downloaded at build time and that it contained the the following:
>
> <html>
> <head><title>404 Not Found</title></head>
> <body bgcolor="white">
> <center><h1>404 Not Found</h1></center>
> <hr><center>nginx</center>
> </body>
> </html>
>
> which is an HTML error message to the effect that the file is missing
> (from the web server).
>
>
> The script sbt-launch-lib.bash contains the following lines which
> determine where the file sbt-launch.jar is downloaded from:
>
> acquire_sbt_jar () {
>   SBT_VERSION=`awk -F "=" '/sbt\.version/ {print $2}'
> ./project/build.properties`
>   URL1=
> http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
>   URL2=
> http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
>   JAR=build/sbt-launch-${SBT_VERSION}.jar
>
>
> The script sbt-launch.bash downloads $URL1 first, and incorrectly
> concludes that it succeeded on the basis that the file sbt-launch-0.13.7.jar
> exists (though it contains HTML).
>
> I succeeded in building Spark by:
>
> (1)  Downloading the file sbt-launch-0.13.7.jar from $URL2 and placing it
> in the build directory.
> (2)  Modifying sbt-launch-lib.bash to prevent the download of that file.
> (3)  Restarting the download as I usually would, with "SPARK_HIVE=true
> SPARK_HADOOP_VERSION=2.5.1 sbt/sbt clean assembly"
>
>
> I think a lot of people will be confused by this.  Probably someone should
> do some of the following:
>
> (1)  Delete $URL1 and all references, or replace it with the
> correct/current URL which points to the sbt-launch.jar(s).
> (2)  Modify sbt-launch-lib.bash, so that it will not conclude that the
> download of sbt-launch.jar succeeded, when the data returned is an HTML
> error message.
>
>
> Let me know if this is not clear, I will gladly explain in more detail or
> with more clarity, if needed.
>
> -Bryce Lobdell
>
>
> A transcript of my console is below:
>
>
>
>
> xxxxxxxx@ip-xx-xxx-xx-xxx:~/spark/spark-1.4.0$ SPARK_HIVE=true
> SPARK_HADOOP_VERSION=2.5.1 sbt/sbt clean assembly
> NOTE: The sbt/sbt script has been relocated to build/sbt.
>       Please update references to point to the new location.
>
>       Invoking 'build/sbt clean assembly' now ...
>
> Using /usr/lib/jvm/java-7-openjdk-amd64/ as default JAVA_HOME.
> Note, this will be overridden by -java-home if it is set.
> Attempting to fetch sbt
> Launching sbt from build/sbt-launch-0.13.7.jar
> *Error: Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar*
> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0$ cd build/
> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls
> mvn  sbt  sbt-launch-0.13.7.jar  sbt-launch-lib.bash
> *inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ unzip -l
> sbt-launch-0.13.7.jar*
> *Archive:  sbt-launch-0.13.7.jar*
> *  End-of-central-directory signature not found.  Either this file is not*
> *  a zipfile, or it constitutes one disk of a multi-part archive.  In the*
> *  latter case the central directory and zipfile comment will be found on*
> *  the last disk(s) of this archive.*
> unzip:  cannot find zipfile directory in one of sbt-launch-0.13.7.jar or
>         sbt-launch-0.13.7.jar.zip, and cannot find
> sbt-launch-0.13.7.jar.ZIP, period.
> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls
> mvn  sbt  sbt-launch-0.13.7.jar  sbt-launch-lib.bash
> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls -l
> total 28
> -rwxr-xr-x 1 inquidia inquidia 5384 Jun  3 01:07 mvn
> -rwxr-xr-x 1 inquidia inquidia 5395 Jun  3 01:07 sbt
> -rw-rw-r-- 1 inquidia inquidia  162 Aug  7 20:24 sbt-launch-0.13.7.jar
> -rwxr-xr-x 1 inquidia inquidia 5285 Jun  3 01:07 sbt-launch-lib.bash
> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls -l
> total 28
> -rwxr-xr-x 1 inquidia inquidia 5384 Jun  3 01:07 mvn
> -rwxr-xr-x 1 inquidia inquidia 5395 Jun  3 01:07 sbt
> -rw-rw-r-- 1 inquidia inquidia  *162 *Aug  7 20:24 sbt-launch-0.13.7.jar
> -rwxr-xr-x 1 inquidia inquidia 5285 Jun  3 01:07 sbt-launch-lib.bash
> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ cat
> sbt-launch-0.13.7.jar
> *<html>*
> *<head><title>404 Not Found</title></head>*
> *<body bgcolor="white">*
> *<center><h1>404 Not Found</h1></center>*
> *<hr><center>nginx</center>*
> *</body>*
> *</html>*
>
>
>
>

Mime
View raw message