spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: spark config
Date Fri, 07 Aug 2015 22:23:35 GMT
Looks like Sean fixed it:

[SPARK-9633] [BUILD] SBT download locations outdated; need an update

Cheers

On Fri, Aug 7, 2015 at 3:22 PM, Dean Wampler <deanwampler@gmail.com> wrote:

> That's the correct URL. Recent change? The last time I looked, earlier
> this week, it still had the obsolete artifactory URL for URL1 ;)
>
> Dean Wampler, Ph.D.
> Author: Programming Scala, 2nd Edition
> <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
> Typesafe <http://typesafe.com>
> @deanwampler <http://twitter.com/deanwampler>
> http://polyglotprogramming.com
>
> On Fri, Aug 7, 2015 at 5:19 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> In master branch, build/sbt-launch-lib.bash has the following:
>>
>>   URL1=
>> https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
>>
>> I verified that the following exists:
>>
>>
>> https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.7/#sbt-launch.jar
>>
>> FYI
>>
>> On Fri, Aug 7, 2015 at 2:08 PM, Bryce Lobdell <lobdellb@gmail.com> wrote:
>>
>>>
>>> I Recently downloaded spark package 1.4.0:
>>>
>>> A build of Spark with "sbt/sbt clean assembly" failed with message
>>> "Error: Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar"
>>>
>>> Upon investigation I figured out that "sbt-launch-0.13.7.jar" is
>>> downloaded at build time and that it contained the the following:
>>>
>>> <html>
>>> <head><title>404 Not Found</title></head>
>>> <body bgcolor="white">
>>> <center><h1>404 Not Found</h1></center>
>>> <hr><center>nginx</center>
>>> </body>
>>> </html>
>>>
>>> which is an HTML error message to the effect that the file is missing
>>> (from the web server).
>>>
>>>
>>> The script sbt-launch-lib.bash contains the following lines which
>>> determine where the file sbt-launch.jar is downloaded from:
>>>
>>> acquire_sbt_jar () {
>>>   SBT_VERSION=`awk -F "=" '/sbt\.version/ {print $2}'
>>> ./project/build.properties`
>>>   URL1=
>>> http://typesafe.artifactoryonline.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
>>>   URL2=
>>> http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/${SBT_VERSION}/sbt-launch.jar
>>>   JAR=build/sbt-launch-${SBT_VERSION}.jar
>>>
>>>
>>> The script sbt-launch.bash downloads $URL1 first, and incorrectly
>>> concludes that it succeeded on the basis that the file sbt-launch-0.13.7.jar
>>> exists (though it contains HTML).
>>>
>>> I succeeded in building Spark by:
>>>
>>> (1)  Downloading the file sbt-launch-0.13.7.jar from $URL2 and placing
>>> it in the build directory.
>>> (2)  Modifying sbt-launch-lib.bash to prevent the download of that file.
>>> (3)  Restarting the download as I usually would, with "SPARK_HIVE=true
>>> SPARK_HADOOP_VERSION=2.5.1 sbt/sbt clean assembly"
>>>
>>>
>>> I think a lot of people will be confused by this.  Probably someone
>>> should do some of the following:
>>>
>>> (1)  Delete $URL1 and all references, or replace it with the
>>> correct/current URL which points to the sbt-launch.jar(s).
>>> (2)  Modify sbt-launch-lib.bash, so that it will not conclude that the
>>> download of sbt-launch.jar succeeded, when the data returned is an HTML
>>> error message.
>>>
>>>
>>> Let me know if this is not clear, I will gladly explain in more detail
>>> or with more clarity, if needed.
>>>
>>> -Bryce Lobdell
>>>
>>>
>>> A transcript of my console is below:
>>>
>>>
>>>
>>>
>>> xxxxxxxx@ip-xx-xxx-xx-xxx:~/spark/spark-1.4.0$ SPARK_HIVE=true
>>> SPARK_HADOOP_VERSION=2.5.1 sbt/sbt clean assembly
>>> NOTE: The sbt/sbt script has been relocated to build/sbt.
>>>       Please update references to point to the new location.
>>>
>>>       Invoking 'build/sbt clean assembly' now ...
>>>
>>> Using /usr/lib/jvm/java-7-openjdk-amd64/ as default JAVA_HOME.
>>> Note, this will be overridden by -java-home if it is set.
>>> Attempting to fetch sbt
>>> Launching sbt from build/sbt-launch-0.13.7.jar
>>> *Error: Invalid or corrupt jarfile build/sbt-launch-0.13.7.jar*
>>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0$ cd build/
>>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls
>>> mvn  sbt  sbt-launch-0.13.7.jar  sbt-launch-lib.bash
>>> *inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ unzip -l
>>> sbt-launch-0.13.7.jar*
>>> *Archive:  sbt-launch-0.13.7.jar*
>>> *  End-of-central-directory signature not found.  Either this file is
>>> not*
>>> *  a zipfile, or it constitutes one disk of a multi-part archive.  In
>>> the*
>>> *  latter case the central directory and zipfile comment will be found
>>> on*
>>> *  the last disk(s) of this archive.*
>>> unzip:  cannot find zipfile directory in one of sbt-launch-0.13.7.jar or
>>>         sbt-launch-0.13.7.jar.zip, and cannot find
>>> sbt-launch-0.13.7.jar.ZIP, period.
>>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls
>>> mvn  sbt  sbt-launch-0.13.7.jar  sbt-launch-lib.bash
>>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls -l
>>> total 28
>>> -rwxr-xr-x 1 inquidia inquidia 5384 Jun  3 01:07 mvn
>>> -rwxr-xr-x 1 inquidia inquidia 5395 Jun  3 01:07 sbt
>>> -rw-rw-r-- 1 inquidia inquidia  162 Aug  7 20:24 sbt-launch-0.13.7.jar
>>> -rwxr-xr-x 1 inquidia inquidia 5285 Jun  3 01:07 sbt-launch-lib.bash
>>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ ls -l
>>> total 28
>>> -rwxr-xr-x 1 inquidia inquidia 5384 Jun  3 01:07 mvn
>>> -rwxr-xr-x 1 inquidia inquidia 5395 Jun  3 01:07 sbt
>>> -rw-rw-r-- 1 inquidia inquidia  *162 *Aug  7 20:24 sbt-launch-0.13.7.jar
>>> -rwxr-xr-x 1 inquidia inquidia 5285 Jun  3 01:07 sbt-launch-lib.bash
>>> inquidia@ip-10-102-69-107:~/spark/spark-1.4.0/build$ cat
>>> sbt-launch-0.13.7.jar
>>> *<html>*
>>> *<head><title>404 Not Found</title></head>*
>>> *<body bgcolor="white">*
>>> *<center><h1>404 Not Found</h1></center>*
>>> *<hr><center>nginx</center>*
>>> *</body>*
>>> *</html>*
>>>
>>>
>>>
>>>
>>
>

Mime
View raw message