spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dean Wampler <deanwamp...@gmail.com>
Subject Re: spark config
Date Fri, 07 Aug 2015 22:22:22 GMT
That's the correct URL. Recent change? The last time I looked, earlier this
week, it still had the obsolete artifactory URL for URL1 ;)

Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com

On Fri, Aug 7, 2015 at 5:19 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> In master branch, build/sbt-launch-lib.bash has the following:
>
>   URL1=
> https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
>
> I verified that the following exists:
>
>
> https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.7/#sbt-launch.jar
>
> FYI
>
> On Fri, Aug 7, 2015 at 2:08 PM, Bryce Lobdell <lobdellb@gmail.com> wrote:
>
>>
>> I Recently downloaded spark package 1.4.0:
>>
>> A build of Spark with "sbt/sbt clean assembly" failed with message
>> "Error: Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar"
>>
>> Upon investigation I figured out that "sbt-launch-0.13.7.jar" is
>> downloaded at build time and that it contained the the following:
>>
>> <html>
>> <head><title>404 Not Found</title></head>
>> <body bgcolor="white">
>> <center><h1>404 Not Found</h1></center>
>> <hr><center>nginx</center>
>> </body>
>> </html>
>>
>> which is an HTML error message to the effect that the file is missing
>> (from the web server).
>>
>>
>> The script sbt-launch-lib.bash contains the following lines which
>> determine where the file sbt-launch.jar is downloaded from:
>>
>> acquire_sbt_jar () {
>>   SBT_VERSION=`awk -F "=" '/sbt\.version/ {print $2}'
>> ./project/build.properties`
>>   URL1=
>> http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
>>   URL2=
>> http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
>>   JAR=build/sbt-launch-${SBT_VERSION}.jar
>>
>>
>> The script sbt-launch.bash downloads $URL1 first, and incorrectly
>> concludes that it succeeded on the basis that the file sbt-launch-0.13.7.jar
>> exists (though it contains HTML).
>>
>> I succeeded in building Spark by:
>>
>> (1)  Downloading the file sbt-launch-0.13.7.jar from $URL2 and placing
>> it in the build directory.
>> (2)  Modifying sbt-launch-lib.bash to prevent the download of that file.
>> (3)  Restarting the download as I usually would, with "SPARK_HIVE=true
>> SPARK_HADOOP_VERSION=2.5.1 sbt/sbt clean assembly"
>>
>>
>> I think a lot of people will be confused by this.  Probably someone
>> should do some of the following:
>>
>> (1)  Delete $URL1 and all references, or replace it with the
>> correct/current URL which points to the sbt-launch.jar(s).
>> (2)  Modify sbt-launch-lib.bash, so that it will not conclude that the
>> download of sbt-launch.jar succeeded, when the data returned is an HTML
>> error message.
>>
>>
>> Let me know if this is not clear, I will gladly explain in more detail or
>> with more clarity, if needed.
>>
>> -Bryce Lobdell
>>
>>
>> A transcript of my console is below:
>>
>>
>>
>>
>> xxxxxxxx@ip-xx-xxx-xx-xxx:~/spark/spark-1.4.0$ SPARK_HIVE=true
>> SPARK_HADOOP_VERSION=2.5.1 sbt/sbt clean assembly
>> NOTE: The sbt/sbt script has been relocated to build/sbt.
>>       Please update references to point to the new location.
>>
>>       Invoking 'build/sbt clean assembly' now ...
>>
>> Using /usr/lib/jvm/java-7-openjdk-amd64/ as default JAVA_HOME.
>> Note, this will be overridden by -java-home if it is set.
>> Attempting to fetch sbt
>> Launching sbt from build/sbt-launch-0.13.7.jar
>> *Error: Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar*
>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0$ cd build/
>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls
>> mvn  sbt  sbt-launch-0.13.7.jar  sbt-launch-lib.bash
>> *inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ unzip -l
>> sbt-launch-0.13.7.jar*
>> *Archive:  sbt-launch-0.13.7.jar*
>> *  End-of-central-directory signature not found.  Either this file is not*
>> *  a zipfile, or it constitutes one disk of a multi-part archive.  In the*
>> *  latter case the central directory and zipfile comment will be found on*
>> *  the last disk(s) of this archive.*
>> unzip:  cannot find zipfile directory in one of sbt-launch-0.13.7.jar or
>>         sbt-launch-0.13.7.jar.zip, and cannot find
>> sbt-launch-0.13.7.jar.ZIP, period.
>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls
>> mvn  sbt  sbt-launch-0.13.7.jar  sbt-launch-lib.bash
>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls -l
>> total 28
>> -rwxr-xr-x 1 inquidia inquidia 5384 Jun  3 01:07 mvn
>> -rwxr-xr-x 1 inquidia inquidia 5395 Jun  3 01:07 sbt
>> -rw-rw-r-- 1 inquidia inquidia  162 Aug  7 20:24 sbt-launch-0.13.7.jar
>> -rwxr-xr-x 1 inquidia inquidia 5285 Jun  3 01:07 sbt-launch-lib.bash
>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls -l
>> total 28
>> -rwxr-xr-x 1 inquidia inquidia 5384 Jun  3 01:07 mvn
>> -rwxr-xr-x 1 inquidia inquidia 5395 Jun  3 01:07 sbt
>> -rw-rw-r-- 1 inquidia inquidia  *162 *Aug  7 20:24 sbt-launch-0.13.7.jar
>> -rwxr-xr-x 1 inquidia inquidia 5285 Jun  3 01:07 sbt-launch-lib.bash
>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ cat
>> sbt-launch-0.13.7.jar
>> *<html>*
>> *<head><title>404 Not Found</title></head>*
>> *<body bgcolor="white">*
>> *<center><h1>404 Not Found</h1></center>*
>> *<hr><center>nginx</center>*
>> *</body>*
>> *</html>*
>>
>>
>>
>>
>

Mime
View raw message